Sep 16 12:01:51.175395 ip-10-0-133-38 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:01:51.613382 ip-10-0-133-38 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:01:51.613382 ip-10-0-133-38 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:01:51.613382 ip-10-0-133-38 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:01:51.613382 ip-10-0-133-38 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:01:51.613382 ip-10-0-133-38 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:01:51.614641 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.614551    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:01:51.618649 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618633    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:51.618649 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618649    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618652    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618657    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618660    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618663    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618666    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618668    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618671    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618674    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618676    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618680    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618682    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618685    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618687    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618691    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618699    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618702    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618704    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618707    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618709    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:51.618757 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618712    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618714    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618717    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618719    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618722    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618725    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618727    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618730    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618732    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618735    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618737    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618740    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618742    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618746    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618751    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618755    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618758    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618761    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618764    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618767    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:51.619257 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618770    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618772    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618775    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618778    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618780    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618783    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618785    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618788    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618791    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618794    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618797    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618799    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618801    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618804    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618807    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618809    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618812    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618814    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618817    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618819    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:51.619739 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618822    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618825    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618828    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618830    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618833    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618837    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618840    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618843    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618845    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618848    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618851    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618853    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618855    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618858    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618861    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618863    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618866    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618868    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618871    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:51.620237 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618874    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618877    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618881    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618885    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618888    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618890    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618893    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618895    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618898    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618900    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618903    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618906    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618910    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618914    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618917    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618919    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618922    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618925    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.618927    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:51.620729 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619628    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619635    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619638    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619641    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619643    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619647    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619650    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619652    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619655    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619658    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619660    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619663    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619665    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619668    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619671    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619673    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619677    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619680    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619683    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619686    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:51.621213 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619688    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619691    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619694    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619696    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619699    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619701    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619704    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619707    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619710    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619712    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619715    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619717    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619720    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619723    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619726    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619729    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619732    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619735    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619737    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:51.621746 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619740    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619743    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619746    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619748    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619751    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619754    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619757    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619760    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619764    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619768    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619772    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619775    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619778    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619781    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619784    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619786    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619789    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619791    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619794    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619796    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:51.622228 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619799    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619802    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619805    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619807    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619810    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619813    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619816    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619819    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619821    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619825    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619828    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619831    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619833    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619836    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619838    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619841    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619843    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619846    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619848    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619851    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:51.622724 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619853    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619856    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619859    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619862    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619865    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619867    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619870    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619872    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619874    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619878    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619882    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619884    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619887    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619890    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619893    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619896    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619898    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619901    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619904    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:51.623231 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.619907    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620286    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620291    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620295    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620297    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620300    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620303    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620306    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620309    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620312    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620314    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620317    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620320    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620322    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620325    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620328    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620331    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620334    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620337    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620339    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:51.623702 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620342    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620344    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620347    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620349    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620351    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620354    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620356    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620359    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620361    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620364    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620367    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620369    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620372    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620375    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620377    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620380    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620383    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620385    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620388    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620391    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:51.624220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620394    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620397    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620399    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620402    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620405    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620408    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620412    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620416    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620418    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620432    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620434    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620437    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620440    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620443    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620445    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620448    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620450    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620453    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620456    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620458    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:51.624711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620461    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620463    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620466    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620469    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620472    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620475    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620478    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620483    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620487    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620490    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620493    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620496    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620498    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620501    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620504    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620506    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620510    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620513    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620516    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:51.625209 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620518    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620521    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620524    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620527    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620530    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620533    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620536    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620539    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620542    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620544    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620547    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620550    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620553    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620555    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620558    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620560    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620563    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620565    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620568    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620570    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:51.625696 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.620579    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620658    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620689    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620695    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620700    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620704    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620707    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620711    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620716    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620719    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620722    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620725    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620729    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620732    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620739    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620742    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620745    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620748    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620751    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620756    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620758    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620762    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620765    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620768    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:01:51.626204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620772    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620775    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620779    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620782    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620785    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620788    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620791    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620794    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620798    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620802    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620805    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620808    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620811    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620814    2577 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620817    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620821    2577 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620824    2577 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620827    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620830    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620833    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620837    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620839    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620844    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620847    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620850    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:01:51.626826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620853    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620857    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620860    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620863    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620866    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620869    2577 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620873    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620876    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620879    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620882    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620885    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620888    2577 flags.go:64] FLAG: --help="false"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620891    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620894    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620897    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620901    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620904    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620908    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620910    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620913    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620916    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620919    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620922    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620925    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:01:51.627467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620928    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620931    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620933    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620936    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620939    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620943    2577 flags.go:64] FLAG: --lock-file=""
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620946    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620949    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620952    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620958    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620961    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620964    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620967    2577 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620970    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620973    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620976    2577 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620979    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620983    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620986    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620989    2577 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620993    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620996    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.620998    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621003    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621007    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:01:51.628063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621010    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621013    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621020    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621023    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621026    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621029    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621033    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621039    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621042    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621045    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621048    2577 flags.go:64] FLAG: --port="10250"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621051    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621054    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00bb776fe61cfe878"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621059    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621062    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621077    2577 flags.go:64] FLAG: --register-node="true"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621081    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621085    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621088    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621091    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621094    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621097    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621100    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621104    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621107    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:01:51.628709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621109    2577 flags.go:64] FLAG: --runonce="false"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621112    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621115    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621118    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621121    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621125    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621128    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621132    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621135    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621138    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621141    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621143    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621147    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621150    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621153    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621155    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621161    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621163    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621166    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621171    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621175    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621178    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621181    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621184    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621187    2577 flags.go:64] FLAG: --v="2"
Sep 16 12:01:51.629360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621191    2577 flags.go:64] FLAG: --version="false"
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621195    2577 flags.go:64] FLAG: --vmodule=""
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621199    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.621202    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621328    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621332    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621335    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621338    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621341    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621344    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621347    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621349    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621353    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621356    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621358    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621362    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621366    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621369    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621372    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621374    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:51.630226 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621377    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621380    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621382    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621385    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621388    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621390    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621393    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621397    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621399    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621402    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621404    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621407    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621410    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621412    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621415    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621418    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621422    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621425    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621428    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:51.630860 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621431    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621433    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621436    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621438    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621441    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621444    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621446    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621449    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621451    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621454    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621456    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621459    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621462    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621464    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621467    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621469    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621472    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621474    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621477    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621480    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:51.631364 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621483    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621486    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621489    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621491    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621494    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621496    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621499    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621502    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621505    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621507    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621510    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621512    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621515    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621518    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621521    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621523    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621533    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621536    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621539    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621542    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:51.631853 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621544    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621547    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621550    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621552    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621555    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621558    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621560    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621563    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621567    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621570    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621573    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621576    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621581    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621584    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621586    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621589    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621592    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621595    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621598    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621600    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:51.632380 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621604    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621607    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621610    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.621613    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.622322    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.631671    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.631687    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.632556    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.632564    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632622    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632627    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632631    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632634    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632637    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632640    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:51.632868 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632643    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632646    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632649    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632652    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632655    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632658    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632661    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632667    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632672    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632675    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632678    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632681    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632683    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632686    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632688    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632691    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632694    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632697    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632700    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:51.633269 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632703    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632706    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632708    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632711    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632715    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632718    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632720    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632724    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632726    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632729    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632732    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632734    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632737    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632740    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632742    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632745    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632747    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632750    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632753    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632755    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:51.633737 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632758    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632762    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632764    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632767    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632769    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632772    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632775    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632777    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632780    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632782    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632785    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632788    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632790    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632793    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632796    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632799    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632802    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632805    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632808    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632811    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:51.634295 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632814    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632816    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632819    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632821    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632824    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632826    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632829    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632832    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632835    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632837    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632840    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632842    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632845    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632848    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632851    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632853    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632856    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632858    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632861    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632864    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:51.634780 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632866    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632871    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632874    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632877    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632880    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632883    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632886    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632889    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632892    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632894    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632897    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632900    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632903    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.632905    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.632910    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:01:51.635323 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633022    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633027    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633030    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633032    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633036    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633038    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633041    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633044    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633046    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633049    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633052    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633054    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633058    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633061    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633063    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633078    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633083    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633086    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633089    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633092    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:01:51.635701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633095    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633098    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633100    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633103    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633106    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633109    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633111    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633114    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633116    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633119    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633122    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633125    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633127    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633130    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633133    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633135    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633138    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633142    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633145    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:01:51.636246 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633149    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633152    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633154    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633158    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633161    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633164    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633167    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633170    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633173    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633175    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633178    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633181    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633184    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633187    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633189    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633192    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633194    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633197    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633199    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:01:51.636701 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633203    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633205    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633208    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633211    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633213    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633216    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633218    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633221    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633223    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633226    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633228    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633231    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633233    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633237    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633241    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633243    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633246    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633249    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633252    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:01:51.637183 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633255    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633258    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633261    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633263    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633267    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633269    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633272    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633275    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633278    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633281    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633283    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633286    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633289    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633291    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633294    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633296    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633299    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633302    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633305    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633307    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:01:51.637645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633310    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:01:51.638141 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:51.633313    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:01:51.638141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.633318    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:01:51.638141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.633423    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:01:51.638141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.635731    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:01:51.638141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.636757    2577 server.go:1019] "Starting client certificate rotation"
Sep 16 12:01:51.638141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.636857    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:01:51.638141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.637685    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:01:51.639575 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.639561    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:01:51.640713 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.640698    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:01:51.653193 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.653177    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:01:51.657395 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.657380    2577 log.go:25] "Validated CRI v1 image API"
Sep 16 12:01:51.658263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.658241    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:01:51.660810 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.660786    2577 fs.go:135] Filesystem UUIDs: map[0ab1725b-18e5-4d2a-bdba-6bcef96a95dc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dcc2f04e-b32d-462e-8fb4-9d4e08d58857:/dev/nvme0n1p3]
Sep 16 12:01:51.660869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.660810    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:01:51.666736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.666718    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:01:51.667813 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.667711    2577 manager.go:216] Machine: {Timestamp:2026-09-16 12:01:51.666043638 +0000 UTC m=+0.377559676 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105919 MemoryCapacity:33164464128 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21859a6a116f69e729fb08c62495bb SystemUUID:ec21859a-6a11-6f69-e729-fb08c62495bb BootID:6fe6ae23-54ad-44eb-ae5c-d2c4f6559c63 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582230016 Type:vfs Inodes:4048396 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:7d:3a:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:7d:3a:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:ee:0a:91:a1:9c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164464128 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:01:51.667813 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.667812    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:01:51.667921 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.667893    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:01:51.671329 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.671307    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:01:51.671460 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.671332    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:01:51.672159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.672149    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:01:51.672199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.672161    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:01:51.672199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.672168    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:01:51.672199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.672189    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:01:51.673460 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.673450    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:01:51.678569 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.678557    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:01:51.678679 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.678670    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:01:51.678708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.678702    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:01:51.678736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.678714    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:01:51.678773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.678738    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:01:51.678773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.678752    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:01:51.678773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.678765    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:01:51.679190 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.679178    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:01:51.679241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.679198    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:01:51.681552 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.681529    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:01:51.683805 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.683792    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:01:51.683871 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.683822    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:01:51.685836 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685818    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t45v4"
Sep 16 12:01:51.685889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685879    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:01:51.685927 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685898    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:01:51.685927 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685906    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:01:51.685927 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685913    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:01:51.685927 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685922    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:01:51.686038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685931    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:01:51.686038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685946    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:01:51.686038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685955    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:01:51.686038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685964    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:01:51.686038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685972    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:01:51.686038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.685995    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:01:51.686038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.686001    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:01:51.686901 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.686890    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:01:51.686932 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.686902    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:01:51.687607 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.687581    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:01:51.687976 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.687960    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:01:51.690653 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.690641    2577 server.go:1268] "Started kubelet"
Sep 16 12:01:51.690717 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.690694    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:01:51.691394 ip-10-0-133-38 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:01:51.692364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.691568    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:01:51.692364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.691624    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:01:51.693018 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.693004    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:01:51.693459 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.693447    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:01:51.694025 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.693995    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t45v4"
Sep 16 12:01:51.696526 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.696511    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:01:51.696829 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.696809    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:01:51.700515 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.700500    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:01:51.701167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.701149    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:01:51.701516 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.701494    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:51.702246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702225    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:01:51.702412 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702391    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:01:51.702412 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702412    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:01:51.702559 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702443    2577 factory.go:55] Registering systemd factory
Sep 16 12:01:51.702559 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702460    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:01:51.702656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702561    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:01:51.702656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702570    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:01:51.702850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702836    2577 factory.go:153] Registering CRI-O factory
Sep 16 12:01:51.702923 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702854    2577 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:01:51.702923 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702906    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:01:51.703017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702925    2577 factory.go:103] Registering Raw factory
Sep 16 12:01:51.703017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.702952    2577 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:01:51.705706 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.703372    2577 manager.go:349] Starting recovery of all containers
Sep 16 12:01:51.706296 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.706275    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:51.709453 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.709420    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-38.ec2.internal\" not found" node="ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.714553 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.714419    2577 manager.go:354] Recovery completed
Sep 16 12:01:51.716687 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.716667    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 12:01:51.719778 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.719765    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.721759 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.721743    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.721835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.721777    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.721835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.721792    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.722377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.722359    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:01:51.722377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.722375    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:01:51.722489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.722392    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.722489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.722394    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:01:51.723136 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.723120    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.723196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.723147    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.723196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.723157    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.724581 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.724568    2577 policy_none.go:49] "None policy: Start"
Sep 16 12:01:51.724615 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.724586    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:01:51.724615 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.724596    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:01:51.726663 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.726651    2577 policy_none.go:47] "Start"
Sep 16 12:01:51.756680 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.756666    2577 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.756701    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.756712    2577 server.go:87] "Starting device plugin registration server"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.756970    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.756986    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.757058    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.757141    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.757149    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.757835    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:01:51.783617 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.757906    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:51.804163 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.804132    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:01:51.805338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.805320    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:01:51.805435 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.805367    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:01:51.805435 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.805390    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:01:51.805540 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.805488    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:01:51.807764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.807748    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:51.857645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.857602    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.858689 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.858669    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.858800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.858706    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.858800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.858719    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.858800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.858747    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.868010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.867924    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.868010 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.867949    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-38.ec2.internal\": node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:51.883784 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.883764    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:51.906080 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.906026    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal"]
Sep 16 12:01:51.906200 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.906123    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.907172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.907156    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.907253 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.907184    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.907253 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.907196    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.909452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.909440    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.909584 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.909556    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.909584 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.909585    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.910241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.910225    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.910306 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.910249    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.910306 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.910258    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.910812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.910796    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.910887 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.910822    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.910887 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.910833    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.912935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.912920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.912935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.912941    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:01:51.916156 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.916138    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:01:51.916263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.916165    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:01:51.916263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:51.916174    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:01:51.933714 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.933686    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-38.ec2.internal\" not found" node="ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.939674 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.939654    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-38.ec2.internal\" not found" node="ip-10-0-133-38.ec2.internal"
Sep 16 12:01:51.984893 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:51.984858    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:52.086016 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:52.085974    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:52.103277 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.103253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3571c7ce18de9915aa1b3a13a6c6bc9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-38.ec2.internal\" (UID: \"a3571c7ce18de9915aa1b3a13a6c6bc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.103347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.103283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b94570915550ff53f158095ff319695-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal\" (UID: \"6b94570915550ff53f158095ff319695\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.103347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.103304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b94570915550ff53f158095ff319695-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal\" (UID: \"6b94570915550ff53f158095ff319695\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.186629 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:52.186562    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:52.203908 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.203889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b94570915550ff53f158095ff319695-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal\" (UID: \"6b94570915550ff53f158095ff319695\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.203966 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.203911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b94570915550ff53f158095ff319695-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal\" (UID: \"6b94570915550ff53f158095ff319695\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.203966 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.203929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3571c7ce18de9915aa1b3a13a6c6bc9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-38.ec2.internal\" (UID: \"a3571c7ce18de9915aa1b3a13a6c6bc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.204028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.203974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3571c7ce18de9915aa1b3a13a6c6bc9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-38.ec2.internal\" (UID: \"a3571c7ce18de9915aa1b3a13a6c6bc9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.204028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.203991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b94570915550ff53f158095ff319695-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal\" (UID: \"6b94570915550ff53f158095ff319695\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.204028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.204002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b94570915550ff53f158095ff319695-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal\" (UID: \"6b94570915550ff53f158095ff319695\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.236012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.235991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.242102 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:52.242081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:52.287541 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:52.287510    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:53.687539 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:53.687496    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:53.788126 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:53.788046    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:53.888600 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:53.888561    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:53.936118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:53.936081    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:01:53.936696 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:53.936247    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:01:53.936696 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:53.936296    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:01:53.989641 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:53.989614    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:53.998397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:53.998353    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 11:56:52 +0000 UTC" deadline="2028-06-05 06:07:19.445740542 +0000 UTC"
Sep 16 12:01:53.998397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:53.998393    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15066h5m25.447350145s"
Sep 16 12:01:54.090553 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:54.090487    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:54.143059 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.143036    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:54.148052 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:54.148017    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3571c7ce18de9915aa1b3a13a6c6bc9.slice/crio-f21d4cfe1e16f62f9029063e7a1463a681dede3e301f53adc6bee28975c9a2e7 WatchSource:0}: Error finding container f21d4cfe1e16f62f9029063e7a1463a681dede3e301f53adc6bee28975c9a2e7: Status 404 returned error can't find the container with id f21d4cfe1e16f62f9029063e7a1463a681dede3e301f53adc6bee28975c9a2e7
Sep 16 12:01:54.148277 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:54.148254    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b94570915550ff53f158095ff319695.slice/crio-1ba35e6dd1e58b0e4044d90d0d486aabbe6b8e25a821357f8c916b6baec51c9d WatchSource:0}: Error finding container 1ba35e6dd1e58b0e4044d90d0d486aabbe6b8e25a821357f8c916b6baec51c9d: Status 404 returned error can't find the container with id 1ba35e6dd1e58b0e4044d90d0d486aabbe6b8e25a821357f8c916b6baec51c9d
Sep 16 12:01:54.151660 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.151646    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:01:54.190923 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:54.190890    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:54.291421 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:54.291381    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-38.ec2.internal\" not found"
Sep 16 12:01:54.309413 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.309393    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:54.402109 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.402027    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:54.414644 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.414622    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:01:54.415776 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.415763    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal"
Sep 16 12:01:54.424295 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.424278    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:01:54.798103 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.797955    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:54.979850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.979818    2577 apiserver.go:52] "Watching apiserver"
Sep 16 12:01:54.989358 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.989330    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:01:54.990694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.990664    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-k8r7h","openshift-network-diagnostics/network-check-target-92vxd","openshift-network-operator/iptables-alerter-hsrvj","kube-system/global-pull-secret-syncer-lc92b","openshift-dns/node-resolver-2s4xk","openshift-image-registry/node-ca-cvcd6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal","openshift-multus/multus-j749k","openshift-multus/network-metrics-daemon-lnxvx","openshift-ovn-kubernetes/ovnkube-node-8xtlh","kube-system/konnectivity-agent-h24wk","kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5","openshift-cluster-node-tuning-operator/tuned-zwv6n"]
Sep 16 12:01:54.993370 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.993295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:54.995607 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.995558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:54.996933 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.996646    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:01:54.996933 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.996655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:01:54.996933 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.996653    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:01:54.996933 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.996790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rbr8r\""
Sep 16 12:01:54.998242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.997983    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:01:54.998242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.997994    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:01:54.998242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.998061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:01:54.998242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.998108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:01:54.998242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.998109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:01:54.998242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.998173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:54.998242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.998245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:01:54.998865 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:54.998784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kk577\""
Sep 16 12:01:55.000759 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.000678    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:01:55.001222 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.001010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.001485 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.001361    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:01:55.002238 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.001938    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:01:55.002238 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.002174    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:01:55.002354 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.002246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:01:55.002615 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.002598    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tf62w\""
Sep 16 12:01:55.004967 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.004949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.006859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.006839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.007594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.007521    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:01:55.007594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.007535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wkfhq\""
Sep 16 12:01:55.008061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.008043    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:01:55.009252 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.009084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:55.009252 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.009159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:01:55.009612 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.009596    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:01:55.009705 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.009675    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:01:55.009762 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.009725    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:01:55.009807 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.009761    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2n2n7\""
Sep 16 12:01:55.011035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-env-overrides\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011134 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.011134 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-systemd-units\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011134 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-var-lib-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011134 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-log-socket\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-cni-bin\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-cni-netd\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-ovnkube-script-lib\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hp4h\" (UniqueName: \"kubernetes.io/projected/93f1d2fc-894d-4056-907f-b3d3ceabcc44-kube-api-access-4hp4h\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.011310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-kubelet\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-node-log\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h7t\" (UniqueName: \"kubernetes.io/projected/7e561896-785d-4507-b5e1-54a6040be27f-kube-api-access-d5h7t\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-sys-fs\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11230c67-bdbb-48f7-8646-7dbf4544bd02-iptables-alerter-script\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11230c67-bdbb-48f7-8646-7dbf4544bd02-host-slash\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4rhs\" (UniqueName: \"kubernetes.io/projected/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-kube-api-access-t4rhs\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdf83e47-967c-46a9-8e56-ac2a7b147088-dbus\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.011516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-registration-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-device-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-ovn\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9dx6\" (UniqueName: \"kubernetes.io/projected/11230c67-bdbb-48f7-8646-7dbf4544bd02-kube-api-access-n9dx6\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-slash\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-run-netns\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.011985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.011838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e561896-785d-4507-b5e1-54a6040be27f-ovn-node-metrics-cert\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-socket-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-hosts-file\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-tmp-dir\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-systemd\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-ovnkube-config\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdf83e47-967c-46a9-8e56-ac2a7b147088-kubelet-config\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.012485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.012203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-etc-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.014495 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.014406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:01:55.014833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.014650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:55.014833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.014694    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:01:55.014833 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.014704    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:01:55.014833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.014708    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:01:55.015300 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.015274    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xsmws\""
Sep 16 12:01:55.016833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.016813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.019154 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.019137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.019246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.019214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:01:55.019757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.019607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:01:55.019757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.019636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:01:55.019757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.019686    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:01:55.019957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.019816    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-444nq\""
Sep 16 12:01:55.019957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.019921    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:01:55.021497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.021478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.021895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.021876    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:01:55.021999 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.021921    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:01:55.022055 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.022009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vmkrn\""
Sep 16 12:01:55.023787 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.023769    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tdx8w\""
Sep 16 12:01:55.023913 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.023896    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:01:55.034058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.034033    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q2g6p"
Sep 16 12:01:55.041501 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.041479    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q2g6p"
Sep 16 12:01:55.103372 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.103343    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:01:55.111379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.111332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal" event={"ID":"a3571c7ce18de9915aa1b3a13a6c6bc9","Type":"ContainerStarted","Data":"f21d4cfe1e16f62f9029063e7a1463a681dede3e301f53adc6bee28975c9a2e7"}
Sep 16 12:01:55.112397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11230c67-bdbb-48f7-8646-7dbf4544bd02-iptables-alerter-script\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.112528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.112528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-kubelet\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.112528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h7t\" (UniqueName: \"kubernetes.io/projected/7e561896-785d-4507-b5e1-54a6040be27f-kube-api-access-d5h7t\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.112528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-var-lib-kubelet\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.112528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsctk\" (UniqueName: \"kubernetes.io/projected/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-kube-api-access-vsctk\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-hostroot\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-kubelet\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-multus-certs\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gdf\" (UniqueName: \"kubernetes.io/projected/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-kube-api-access-z7gdf\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdf83e47-967c-46a9-8e56-ac2a7b147088-dbus\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-sys\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal" event={"ID":"6b94570915550ff53f158095ff319695","Type":"ContainerStarted","Data":"1ba35e6dd1e58b0e4044d90d0d486aabbe6b8e25a821357f8c916b6baec51c9d"}
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-kubelet\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdf83e47-967c-46a9-8e56-ac2a7b147088-dbus\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:55.112796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjn8c\" (UniqueName: \"kubernetes.io/projected/78b0b391-4e63-4177-a34c-c226ca5bc797-kube-api-access-pjn8c\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-registration-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-kubernetes\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-lib-modules\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-registration-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ea27a32-13e1-41d9-bb3d-11620b48445d-tmp\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-cni-bin\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9dx6\" (UniqueName: \"kubernetes.io/projected/11230c67-bdbb-48f7-8646-7dbf4544bd02-kube-api-access-n9dx6\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11230c67-bdbb-48f7-8646-7dbf4544bd02-iptables-alerter-script\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.112994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-netns\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-slash\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e561896-785d-4507-b5e1-54a6040be27f-ovn-node-metrics-cert\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-socket-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-slash\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-k8s-cni-cncf-io\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.113389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-conf-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-systemd\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdf83e47-967c-46a9-8e56-ac2a7b147088-kubelet-config\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-tuned\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdf83e47-967c-46a9-8e56-ac2a7b147088-kubelet-config\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-systemd\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-etc-selinux\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-daemon-config\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-socket-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-modprobe-d\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-run\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-systemd-units\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-var-lib-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113584    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:01:55.114247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-log-socket\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-systemd-units\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-cni-bin\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-var-lib-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-cni-netd\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hp4h\" (UniqueName: \"kubernetes.io/projected/93f1d2fc-894d-4056-907f-b3d3ceabcc44-kube-api-access-4hp4h\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-log-socket\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11230c67-bdbb-48f7-8646-7dbf4544bd02-host-slash\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-cni-bin\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-node-log\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-cni-netd\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-sys-fs\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4rhs\" (UniqueName: \"kubernetes.io/projected/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-kube-api-access-t4rhs\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-systemd\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11230c67-bdbb-48f7-8646-7dbf4544bd02-host-slash\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-node-log\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-sys-fs\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.115138 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.113940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aba3353-392c-4424-a645-ec6ea7d72619-konnectivity-ca\") pod \"konnectivity-agent-h24wk\" (UID: \"0aba3353-392c-4424-a645-ec6ea7d72619\") " pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-cni-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-host\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-etc-kubernetes\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-device-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysconfig\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysctl-d\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-system-cni-dir\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-device-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cnibin\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aba3353-392c-4424-a645-ec6ea7d72619-agent-certs\") pod \"konnectivity-agent-h24wk\" (UID: \"0aba3353-392c-4424-a645-ec6ea7d72619\") " pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-socket-dir-parent\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-ovn\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.116095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysctl-conf\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mph\" (UniqueName: \"kubernetes.io/projected/2ea27a32-13e1-41d9-bb3d-11620b48445d-kube-api-access-88mph\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-run-ovn\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-cnibin\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-cni-multus\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-run-netns\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-hosts-file\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-tmp-dir\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.114538    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db0b38f7-df0b-4486-9151-97c4fe311900-serviceca\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjx5k\" (UniqueName: \"kubernetes.io/projected/db0b38f7-df0b-4486-9151-97c4fe311900-kube-api-access-fjx5k\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-ovnkube-config\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.114670    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret podName:bdf83e47-967c-46a9-8e56-ac2a7b147088 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:55.614646961 +0000 UTC m=+3.026663880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret") pod "global-pull-secret-syncer-lc92b" (UID: "bdf83e47-967c-46a9-8e56-ac2a7b147088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-hosts-file\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-system-cni-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.118029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-cni-binary-copy\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-host-run-netns\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-etc-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-env-overrides\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-tmp-dir\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db0b38f7-df0b-4486-9151-97c4fe311900-host\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.114974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-ovnkube-script-lib\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.115030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f1d2fc-894d-4056-907f-b3d3ceabcc44-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.115125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e561896-785d-4507-b5e1-54a6040be27f-etc-openvswitch\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.115149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-ovnkube-config\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.115189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-env-overrides\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.115495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e561896-785d-4507-b5e1-54a6040be27f-ovnkube-script-lib\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.118833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.117350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e561896-785d-4507-b5e1-54a6040be27f-ovn-node-metrics-cert\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.121892 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.121871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h7t\" (UniqueName: \"kubernetes.io/projected/7e561896-785d-4507-b5e1-54a6040be27f-kube-api-access-d5h7t\") pod \"ovnkube-node-8xtlh\" (UID: \"7e561896-785d-4507-b5e1-54a6040be27f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.122003 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.121905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9dx6\" (UniqueName: \"kubernetes.io/projected/11230c67-bdbb-48f7-8646-7dbf4544bd02-kube-api-access-n9dx6\") pod \"iptables-alerter-hsrvj\" (UID: \"11230c67-bdbb-48f7-8646-7dbf4544bd02\") " pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.122681 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.122652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hp4h\" (UniqueName: \"kubernetes.io/projected/93f1d2fc-894d-4056-907f-b3d3ceabcc44-kube-api-access-4hp4h\") pod \"aws-ebs-csi-driver-node-jn6d5\" (UID: \"93f1d2fc-894d-4056-907f-b3d3ceabcc44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.123751 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.123730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4rhs\" (UniqueName: \"kubernetes.io/projected/4942bb0d-de9b-4af1-8ece-2d67d2a91f51-kube-api-access-t4rhs\") pod \"node-resolver-2s4xk\" (UID: \"4942bb0d-de9b-4af1-8ece-2d67d2a91f51\") " pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.215876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aba3353-392c-4424-a645-ec6ea7d72619-konnectivity-ca\") pod \"konnectivity-agent-h24wk\" (UID: \"0aba3353-392c-4424-a645-ec6ea7d72619\") " pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.215876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-cni-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-host\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-etc-kubernetes\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-host\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-cni-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.215980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysconfig\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysctl-d\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysconfig\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-system-cni-dir\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cnibin\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.216104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aba3353-392c-4424-a645-ec6ea7d72619-agent-certs\") pod \"konnectivity-agent-h24wk\" (UID: \"0aba3353-392c-4424-a645-ec6ea7d72619\") " pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-socket-dir-parent\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysctl-conf\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-system-cni-dir\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mph\" (UniqueName: \"kubernetes.io/projected/2ea27a32-13e1-41d9-bb3d-11620b48445d-kube-api-access-88mph\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-cnibin\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cnibin\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysctl-d\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-cni-multus\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db0b38f7-df0b-4486-9151-97c4fe311900-serviceca\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-socket-dir-parent\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjx5k\" (UniqueName: \"kubernetes.io/projected/db0b38f7-df0b-4486-9151-97c4fe311900-kube-api-access-fjx5k\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-system-cni-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-cni-binary-copy\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db0b38f7-df0b-4486-9151-97c4fe311900-host\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.216662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-sysctl-conf\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-var-lib-kubelet\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsctk\" (UniqueName: \"kubernetes.io/projected/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-kube-api-access-vsctk\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-hostroot\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-multus-certs\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db0b38f7-df0b-4486-9151-97c4fe311900-host\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gdf\" (UniqueName: \"kubernetes.io/projected/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-kube-api-access-z7gdf\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-sys\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-kubelet\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-system-cni-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-kubelet\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-cnibin\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-hostroot\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-cni-multus\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-multus-certs\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-etc-kubernetes\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-var-lib-kubelet\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.217457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.216521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aba3353-392c-4424-a645-ec6ea7d72619-konnectivity-ca\") pod \"konnectivity-agent-h24wk\" (UID: \"0aba3353-392c-4424-a645-ec6ea7d72619\") " pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjn8c\" (UniqueName: \"kubernetes.io/projected/78b0b391-4e63-4177-a34c-c226ca5bc797-kube-api-access-pjn8c\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-cni-binary-copy\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-kubernetes\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-lib-modules\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ea27a32-13e1-41d9-bb3d-11620b48445d-tmp\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-cni-bin\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db0b38f7-df0b-4486-9151-97c4fe311900-serviceca\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-netns\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-k8s-cni-cncf-io\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-conf-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-tuned\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-daemon-config\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-modprobe-d\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-run\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.218421 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.217451    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-systemd\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-sys\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.217625    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs podName:78b0b391-4e63-4177-a34c-c226ca5bc797 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:55.717604194 +0000 UTC m=+3.129621084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs") pod "network-metrics-daemon-lnxvx" (UID: "78b0b391-4e63-4177-a34c-c226ca5bc797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-k8s-cni-cncf-io\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-lib-modules\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-run-netns\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-kubernetes\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217928    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-modprobe-d\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-conf-dir\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.217980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-run\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.218029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-systemd\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.218123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.218181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-host-var-lib-cni-bin\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.218450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-multus-daemon-config\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.219271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.219204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aba3353-392c-4424-a645-ec6ea7d72619-agent-certs\") pod \"konnectivity-agent-h24wk\" (UID: \"0aba3353-392c-4424-a645-ec6ea7d72619\") " pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.220345 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.220316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ea27a32-13e1-41d9-bb3d-11620b48445d-etc-tuned\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.220449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.220360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ea27a32-13e1-41d9-bb3d-11620b48445d-tmp\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.223269 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.223248    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:55.223365 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.223276    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:55.223365 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.223292    2577 projected.go:196] Error preparing data for projected volume kube-api-access-trpjm for pod openshift-network-diagnostics/network-check-target-92vxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:55.223468 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.223369    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm podName:f59f7d0f-6700-43b7-b5f5-726f22dc03ea nodeName:}" failed. No retries permitted until 2026-09-16 12:01:55.723351421 +0000 UTC m=+3.135368312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-trpjm" (UniqueName: "kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm") pod "network-check-target-92vxd" (UID: "f59f7d0f-6700-43b7-b5f5-726f22dc03ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:55.228903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.228854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsctk\" (UniqueName: \"kubernetes.io/projected/39d4c6d9-8b6c-4df8-87b1-0feabaff667f-kube-api-access-vsctk\") pod \"multus-additional-cni-plugins-k8r7h\" (UID: \"39d4c6d9-8b6c-4df8-87b1-0feabaff667f\") " pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.229938 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.229783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gdf\" (UniqueName: \"kubernetes.io/projected/d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e-kube-api-access-z7gdf\") pod \"multus-j749k\" (UID: \"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e\") " pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.230895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.230868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjn8c\" (UniqueName: \"kubernetes.io/projected/78b0b391-4e63-4177-a34c-c226ca5bc797-kube-api-access-pjn8c\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:55.231259 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.231239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjx5k\" (UniqueName: \"kubernetes.io/projected/db0b38f7-df0b-4486-9151-97c4fe311900-kube-api-access-fjx5k\") pod \"node-ca-cvcd6\" (UID: \"db0b38f7-df0b-4486-9151-97c4fe311900\") " pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.231340 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.231313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mph\" (UniqueName: \"kubernetes.io/projected/2ea27a32-13e1-41d9-bb3d-11620b48445d-kube-api-access-88mph\") pod \"tuned-zwv6n\" (UID: \"2ea27a32-13e1-41d9-bb3d-11620b48445d\") " pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.307583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.307478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:01:55.316903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.316876    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5"
Sep 16 12:01:55.327728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.327701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hsrvj"
Sep 16 12:01:55.334388 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.334361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8r7h"
Sep 16 12:01:55.342141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.342116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2s4xk"
Sep 16 12:01:55.350195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.350170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cvcd6"
Sep 16 12:01:55.355781 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.355754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zwv6n"
Sep 16 12:01:55.366414 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.366395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:01:55.370996 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.370975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j749k"
Sep 16 12:01:55.478714 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.478681    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:01:55.619743 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.619652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:55.619896 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.619815    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:55.619896 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.619885    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret podName:bdf83e47-967c-46a9-8e56-ac2a7b147088 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:56.619865202 +0000 UTC m=+4.031882078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret") pod "global-pull-secret-syncer-lc92b" (UID: "bdf83e47-967c-46a9-8e56-ac2a7b147088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:55.720878 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.720849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:55.721035 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.720985    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:55.721085 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.721061    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs podName:78b0b391-4e63-4177-a34c-c226ca5bc797 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:56.721041058 +0000 UTC m=+4.133057931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs") pod "network-metrics-daemon-lnxvx" (UID: "78b0b391-4e63-4177-a34c-c226ca5bc797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:55.821513 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:55.821482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:55.821683 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.821630    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:55.821683 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.821653    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:55.821683 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.821662    2577 projected.go:196] Error preparing data for projected volume kube-api-access-trpjm for pod openshift-network-diagnostics/network-check-target-92vxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:55.821788 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:55.821713    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm podName:f59f7d0f-6700-43b7-b5f5-726f22dc03ea nodeName:}" failed. No retries permitted until 2026-09-16 12:01:56.8216963 +0000 UTC m=+4.233713189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-trpjm" (UniqueName: "kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm") pod "network-check-target-92vxd" (UID: "f59f7d0f-6700-43b7-b5f5-726f22dc03ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:56.045405 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:56.045290    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:56:55 +0000 UTC" deadline="2028-06-30 06:52:51.550155748 +0000 UTC"
Sep 16 12:01:56.045405 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:56.045351    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15666h50m55.504807494s"
Sep 16 12:01:56.144093 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.143948    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f1d2fc_894d_4056_907f_b3d3ceabcc44.slice/crio-53ecc49c7ca52c0da18fc3b90c226527d7e8b7c9113a8c033087334cd17a6639 WatchSource:0}: Error finding container 53ecc49c7ca52c0da18fc3b90c226527d7e8b7c9113a8c033087334cd17a6639: Status 404 returned error can't find the container with id 53ecc49c7ca52c0da18fc3b90c226527d7e8b7c9113a8c033087334cd17a6639
Sep 16 12:01:56.145078 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.145040    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25ca5d8_1a18_4596_bc68_11a5ea6d3a6e.slice/crio-854311b1eb19846eec068f6a5a1f3d658023f03fe454ac399acf6b6eb81d0269 WatchSource:0}: Error finding container 854311b1eb19846eec068f6a5a1f3d658023f03fe454ac399acf6b6eb81d0269: Status 404 returned error can't find the container with id 854311b1eb19846eec068f6a5a1f3d658023f03fe454ac399acf6b6eb81d0269
Sep 16 12:01:56.147181 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.146488    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d4c6d9_8b6c_4df8_87b1_0feabaff667f.slice/crio-88b351c6c6d6753803d20518f277693a0007dd5be603f2f6e1e9c09699c43652 WatchSource:0}: Error finding container 88b351c6c6d6753803d20518f277693a0007dd5be603f2f6e1e9c09699c43652: Status 404 returned error can't find the container with id 88b351c6c6d6753803d20518f277693a0007dd5be603f2f6e1e9c09699c43652
Sep 16 12:01:56.148601 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.148126    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e561896_785d_4507_b5e1_54a6040be27f.slice/crio-0777cea940b872322aabd6d075bdc51f29d21bbb0075ff2d46268ee06716eb52 WatchSource:0}: Error finding container 0777cea940b872322aabd6d075bdc51f29d21bbb0075ff2d46268ee06716eb52: Status 404 returned error can't find the container with id 0777cea940b872322aabd6d075bdc51f29d21bbb0075ff2d46268ee06716eb52
Sep 16 12:01:56.150617 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.150594    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11230c67_bdbb_48f7_8646_7dbf4544bd02.slice/crio-a0ccff1ff17d363b9ba8288c801cce62c25a262d025dc89e302621a70fdc05ec WatchSource:0}: Error finding container a0ccff1ff17d363b9ba8288c801cce62c25a262d025dc89e302621a70fdc05ec: Status 404 returned error can't find the container with id a0ccff1ff17d363b9ba8288c801cce62c25a262d025dc89e302621a70fdc05ec
Sep 16 12:01:56.151716 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.151695    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea27a32_13e1_41d9_bb3d_11620b48445d.slice/crio-f34ec8d4b4362458c9a190c0516fdfe5ad9c83a21ff167a32b234775d0d862db WatchSource:0}: Error finding container f34ec8d4b4362458c9a190c0516fdfe5ad9c83a21ff167a32b234775d0d862db: Status 404 returned error can't find the container with id f34ec8d4b4362458c9a190c0516fdfe5ad9c83a21ff167a32b234775d0d862db
Sep 16 12:01:56.154135 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.154032    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aba3353_392c_4424_a645_ec6ea7d72619.slice/crio-e9d79117d82ad751cfb5dfd5a0e6ac92fb4341cfb7e67f9d0e70206f3db2ec39 WatchSource:0}: Error finding container e9d79117d82ad751cfb5dfd5a0e6ac92fb4341cfb7e67f9d0e70206f3db2ec39: Status 404 returned error can't find the container with id e9d79117d82ad751cfb5dfd5a0e6ac92fb4341cfb7e67f9d0e70206f3db2ec39
Sep 16 12:01:56.154302 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.154212    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0b38f7_df0b_4486_9151_97c4fe311900.slice/crio-0ed09bf3b51277f46e0140029e3f22e48c8bf35f846e054e4e06dcfdf4d87a19 WatchSource:0}: Error finding container 0ed09bf3b51277f46e0140029e3f22e48c8bf35f846e054e4e06dcfdf4d87a19: Status 404 returned error can't find the container with id 0ed09bf3b51277f46e0140029e3f22e48c8bf35f846e054e4e06dcfdf4d87a19
Sep 16 12:01:56.155388 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:01:56.155364    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4942bb0d_de9b_4af1_8ece_2d67d2a91f51.slice/crio-e1d3d566b419e7a4307e16cd6c119613a24638203d615ecdcf3ffda15d14907d WatchSource:0}: Error finding container e1d3d566b419e7a4307e16cd6c119613a24638203d615ecdcf3ffda15d14907d: Status 404 returned error can't find the container with id e1d3d566b419e7a4307e16cd6c119613a24638203d615ecdcf3ffda15d14907d
Sep 16 12:01:56.629673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:56.629561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:56.629843 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.629796    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:56.629910 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.629857    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret podName:bdf83e47-967c-46a9-8e56-ac2a7b147088 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:58.629840099 +0000 UTC m=+6.041856993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret") pod "global-pull-secret-syncer-lc92b" (UID: "bdf83e47-967c-46a9-8e56-ac2a7b147088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:56.730410 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:56.730372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:56.730611 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.730588    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:56.730680 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.730653    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs podName:78b0b391-4e63-4177-a34c-c226ca5bc797 nodeName:}" failed. No retries permitted until 2026-09-16 12:01:58.730635118 +0000 UTC m=+6.142651995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs") pod "network-metrics-daemon-lnxvx" (UID: "78b0b391-4e63-4177-a34c-c226ca5bc797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:56.831151 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:56.831094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:56.831338 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.831272    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:56.831338 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.831290    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:56.831338 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.831302    2577 projected.go:196] Error preparing data for projected volume kube-api-access-trpjm for pod openshift-network-diagnostics/network-check-target-92vxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:56.831486 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:56.831362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm podName:f59f7d0f-6700-43b7-b5f5-726f22dc03ea nodeName:}" failed. No retries permitted until 2026-09-16 12:01:58.831343761 +0000 UTC m=+6.243360638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-trpjm" (UniqueName: "kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm") pod "network-check-target-92vxd" (UID: "f59f7d0f-6700-43b7-b5f5-726f22dc03ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:57.046398 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.046307    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:56:55 +0000 UTC" deadline="2028-04-20 09:29:21.83966363 +0000 UTC"
Sep 16 12:01:57.046398 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.046346    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13965h27m24.793322209s"
Sep 16 12:01:57.106669 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.105840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:57.106669 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:57.105990    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:01:57.106669 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.106410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:57.106669 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:57.106523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:01:57.106669 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.106553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:57.106669 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:57.106639    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:01:57.119400 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.119316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerStarted","Data":"88b351c6c6d6753803d20518f277693a0007dd5be603f2f6e1e9c09699c43652"}
Sep 16 12:01:57.123905 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.123842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal" event={"ID":"a3571c7ce18de9915aa1b3a13a6c6bc9","Type":"ContainerStarted","Data":"22dacf339f2c2c6588b937a63ac7b4a4ceb3363a92a3da5273c2de364b5f7ee8"}
Sep 16 12:01:57.128321 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.127773    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b94570915550ff53f158095ff319695" containerID="1f58f32d55a7e03b1c4c1e3d726858aa99a5bda549a6b87480c8fbd5cd7e0dfe" exitCode=0
Sep 16 12:01:57.128321 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.128084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal" event={"ID":"6b94570915550ff53f158095ff319695","Type":"ContainerDied","Data":"1f58f32d55a7e03b1c4c1e3d726858aa99a5bda549a6b87480c8fbd5cd7e0dfe"}
Sep 16 12:01:57.130306 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.130227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cvcd6" event={"ID":"db0b38f7-df0b-4486-9151-97c4fe311900","Type":"ContainerStarted","Data":"0ed09bf3b51277f46e0140029e3f22e48c8bf35f846e054e4e06dcfdf4d87a19"}
Sep 16 12:01:57.133240 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.132835    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hsrvj" event={"ID":"11230c67-bdbb-48f7-8646-7dbf4544bd02","Type":"ContainerStarted","Data":"a0ccff1ff17d363b9ba8288c801cce62c25a262d025dc89e302621a70fdc05ec"}
Sep 16 12:01:57.135512 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.135460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j749k" event={"ID":"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e","Type":"ContainerStarted","Data":"854311b1eb19846eec068f6a5a1f3d658023f03fe454ac399acf6b6eb81d0269"}
Sep 16 12:01:57.146530 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.143863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5" event={"ID":"93f1d2fc-894d-4056-907f-b3d3ceabcc44","Type":"ContainerStarted","Data":"53ecc49c7ca52c0da18fc3b90c226527d7e8b7c9113a8c033087334cd17a6639"}
Sep 16 12:01:57.146530 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.145876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2s4xk" event={"ID":"4942bb0d-de9b-4af1-8ece-2d67d2a91f51","Type":"ContainerStarted","Data":"e1d3d566b419e7a4307e16cd6c119613a24638203d615ecdcf3ffda15d14907d"}
Sep 16 12:01:57.154186 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.154136    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-38.ec2.internal" podStartSLOduration=3.154118655 podStartE2EDuration="3.154118655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:57.138210913 +0000 UTC m=+4.550227811" watchObservedRunningTime="2026-09-16 12:01:57.154118655 +0000 UTC m=+4.566135551"
Sep 16 12:01:57.160180 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.160155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h24wk" event={"ID":"0aba3353-392c-4424-a645-ec6ea7d72619","Type":"ContainerStarted","Data":"e9d79117d82ad751cfb5dfd5a0e6ac92fb4341cfb7e67f9d0e70206f3db2ec39"}
Sep 16 12:01:57.168808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.168749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwv6n" event={"ID":"2ea27a32-13e1-41d9-bb3d-11620b48445d","Type":"ContainerStarted","Data":"f34ec8d4b4362458c9a190c0516fdfe5ad9c83a21ff167a32b234775d0d862db"}
Sep 16 12:01:57.172904 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:57.172812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"0777cea940b872322aabd6d075bdc51f29d21bbb0075ff2d46268ee06716eb52"}
Sep 16 12:01:58.190182 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:58.189675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal" event={"ID":"6b94570915550ff53f158095ff319695","Type":"ContainerStarted","Data":"a48d4c015ff6bb1073b0fee7bd76f27e2924f1de33ac65057713946e0fc05bcf"}
Sep 16 12:01:58.645207 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:58.645135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:58.645411 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.645272    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:58.645411 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.645338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret podName:bdf83e47-967c-46a9-8e56-ac2a7b147088 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:02.645317528 +0000 UTC m=+10.057334403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret") pod "global-pull-secret-syncer-lc92b" (UID: "bdf83e47-967c-46a9-8e56-ac2a7b147088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:01:58.745839 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:58.745803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:58.746011 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.745975    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:58.746067 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.746038    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs podName:78b0b391-4e63-4177-a34c-c226ca5bc797 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:02.746020246 +0000 UTC m=+10.158037132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs") pod "network-metrics-daemon-lnxvx" (UID: "78b0b391-4e63-4177-a34c-c226ca5bc797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:01:58.846824 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:58.846785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:58.847089 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.847067    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:01:58.847170 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.847100    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:01:58.847170 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.847117    2577 projected.go:196] Error preparing data for projected volume kube-api-access-trpjm for pod openshift-network-diagnostics/network-check-target-92vxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:58.847280 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:58.847211    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm podName:f59f7d0f-6700-43b7-b5f5-726f22dc03ea nodeName:}" failed. No retries permitted until 2026-09-16 12:02:02.847161129 +0000 UTC m=+10.259178025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-trpjm" (UniqueName: "kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm") pod "network-check-target-92vxd" (UID: "f59f7d0f-6700-43b7-b5f5-726f22dc03ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:01:59.106058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:59.105634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:01:59.106058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:59.105664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:01:59.106058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:01:59.105634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:01:59.106058 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:59.105767    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:01:59.106058 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:59.105884    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:01:59.106058 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:01:59.105963    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:01.105354 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:01.105320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:01.105881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:01.105320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:01.105881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:01.105450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:01.105881 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:01.105466    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:01.105881 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:01.105526    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:01.105881 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:01.105687    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:02.680601 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:02.680513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:02.681087 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.680689    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:02.681087 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.680771    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret podName:bdf83e47-967c-46a9-8e56-ac2a7b147088 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:10.680747538 +0000 UTC m=+18.092764429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret") pod "global-pull-secret-syncer-lc92b" (UID: "bdf83e47-967c-46a9-8e56-ac2a7b147088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:02.781370 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:02.781309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:02.781553 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.781465    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:02.781553 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.781530    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs podName:78b0b391-4e63-4177-a34c-c226ca5bc797 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:10.781511661 +0000 UTC m=+18.193528550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs") pod "network-metrics-daemon-lnxvx" (UID: "78b0b391-4e63-4177-a34c-c226ca5bc797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:02.881730 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:02.881688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:02.881951 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.881894    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:02:02.881951 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.881922    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:02:02.881951 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.881936    2577 projected.go:196] Error preparing data for projected volume kube-api-access-trpjm for pod openshift-network-diagnostics/network-check-target-92vxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:02.882176 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:02.882002    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm podName:f59f7d0f-6700-43b7-b5f5-726f22dc03ea nodeName:}" failed. No retries permitted until 2026-09-16 12:02:10.88198196 +0000 UTC m=+18.293998854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-trpjm" (UniqueName: "kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm") pod "network-check-target-92vxd" (UID: "f59f7d0f-6700-43b7-b5f5-726f22dc03ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:03.106693 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:03.106612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:03.106850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:03.106734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:03.106850 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:03.106759    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:03.106850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:03.106801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:03.107002 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:03.106927    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:03.107083 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:03.107043    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:05.105370 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:05.105329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:05.105370 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:05.105366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:05.105970 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:05.105468    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:05.105970 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:05.105546    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:05.105970 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:05.105588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:05.105970 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:05.105685    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:07.105435 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:07.105402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:07.105881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:07.105402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:07.105881 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:07.105518    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:07.105881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:07.105526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:07.105881 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:07.105614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:07.105881 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:07.105717    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:09.106066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:09.105853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:09.106519 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:09.105854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:09.106519 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:09.106171    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:09.106519 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:09.105853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:09.106519 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:09.106262    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:09.106519 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:09.106329    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:10.739985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:10.739939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:10.740500 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.740111    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:10.740500 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.740185    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret podName:bdf83e47-967c-46a9-8e56-ac2a7b147088 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:26.740165416 +0000 UTC m=+34.152182296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret") pod "global-pull-secret-syncer-lc92b" (UID: "bdf83e47-967c-46a9-8e56-ac2a7b147088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:10.841144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:10.841104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:10.841329 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.841255    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:10.841329 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.841327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs podName:78b0b391-4e63-4177-a34c-c226ca5bc797 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:26.841310726 +0000 UTC m=+34.253327600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs") pod "network-metrics-daemon-lnxvx" (UID: "78b0b391-4e63-4177-a34c-c226ca5bc797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:10.942156 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:10.942116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:10.942331 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.942266    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:02:10.942331 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.942283    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:02:10.942331 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.942293    2577 projected.go:196] Error preparing data for projected volume kube-api-access-trpjm for pod openshift-network-diagnostics/network-check-target-92vxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:10.942460 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:10.942352    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm podName:f59f7d0f-6700-43b7-b5f5-726f22dc03ea nodeName:}" failed. No retries permitted until 2026-09-16 12:02:26.942338741 +0000 UTC m=+34.354355619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-trpjm" (UniqueName: "kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm") pod "network-check-target-92vxd" (UID: "f59f7d0f-6700-43b7-b5f5-726f22dc03ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:11.105808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:11.105724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:11.105960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:11.105724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:11.105960 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:11.105860    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:11.105960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:11.105724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:11.106108 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:11.105962    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:11.106108 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:11.106021    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:13.110520 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:13.110484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:13.111005 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:13.110631    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:13.111005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:13.110716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:13.111005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:13.110806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:13.111005 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:13.110861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:13.111005 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:13.110884    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:14.228224 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.228007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cvcd6" event={"ID":"db0b38f7-df0b-4486-9151-97c4fe311900","Type":"ContainerStarted","Data":"4975e23ffc8388bce36159f5d1ea211e06e331c46e409742d28dfec5131439ed"}
Sep 16 12:02:14.229993 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.229952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j749k" event={"ID":"d25ca5d8-1a18-4596-bc68-11a5ea6d3a6e","Type":"ContainerStarted","Data":"aa6e5d439b1c3c083115a1cd2f33a8c4f37d78481bd65fec9563eb1e8d1a3aee"}
Sep 16 12:02:14.231794 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.231770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5" event={"ID":"93f1d2fc-894d-4056-907f-b3d3ceabcc44","Type":"ContainerStarted","Data":"931512f6e763ffe585f6fefc3e28572cbcf72b51a156d333cb39c1a2baede9aa"}
Sep 16 12:02:14.233275 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.233254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2s4xk" event={"ID":"4942bb0d-de9b-4af1-8ece-2d67d2a91f51","Type":"ContainerStarted","Data":"993019f1d2ea7eefeea7be612b8db0cc08a1f1c9a715f0bf5d822666b1f2ba86"}
Sep 16 12:02:14.234808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.234787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h24wk" event={"ID":"0aba3353-392c-4424-a645-ec6ea7d72619","Type":"ContainerStarted","Data":"7cf7ecc1d27e3ccbdfdd66ef6c00989dfffe8ca3a0feab14133f1d7dbec131f2"}
Sep 16 12:02:14.236332 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.236310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zwv6n" event={"ID":"2ea27a32-13e1-41d9-bb3d-11620b48445d","Type":"ContainerStarted","Data":"66339721cffbb57119c63fa34ad485e86ccdaf5ba54283775b8a34feeb0bdd92"}
Sep 16 12:02:14.239151 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.239130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"24bd6d325cc658f2e71d1991913bf7edb2bcb4cd8c9933e45e078c73fbec88c6"}
Sep 16 12:02:14.239245 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.239193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"c43cce98825896acc132ce720ec5b6e44c3c4c45c1f808b56a88050c47fad71f"}
Sep 16 12:02:14.239245 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.239235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"e8b4f9ebcc0d31083fa7c6949af2dc3fcb64bbe8211ff7400202d4dd0fffd880"}
Sep 16 12:02:14.239351 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.239275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"20e58ccb523c8610a536ca0997a2ef8e071fa73434a0a3de5edb03202edb81f8"}
Sep 16 12:02:14.239351 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.239309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"ee9058454a9986cfdf25c3eef2599241e471648cfc96665692f9f803aa46fe52"}
Sep 16 12:02:14.239351 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.239349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"20a55d9d2c229d7db6087289fdc79580c9409955d914c457ef1ba6d8110fd13d"}
Sep 16 12:02:14.240620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.240597    2577 generic.go:356] "Generic (PLEG): container finished" podID="39d4c6d9-8b6c-4df8-87b1-0feabaff667f" containerID="de5eaec8fd09694bbf7e0bce6ce5f96f9a5e67b3c7674aaa5b603a6857562c95" exitCode=0
Sep 16 12:02:14.240697 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.240633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerDied","Data":"de5eaec8fd09694bbf7e0bce6ce5f96f9a5e67b3c7674aaa5b603a6857562c95"}
Sep 16 12:02:14.243478 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.243445    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cvcd6" podStartSLOduration=11.100772876 podStartE2EDuration="21.243431533s" totalImagesPullingTime="10.142658657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.243268687 +0000 UTC m=+21.655285585" watchObservedRunningTime="2026-09-16 12:02:14.243431533 +0000 UTC m=+21.655448427"
Sep 16 12:02:14.243555 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.243506    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-38.ec2.internal" podStartSLOduration=20.243502556 podStartE2EDuration="20.243502556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:01:58.215170175 +0000 UTC m=+5.627187072" watchObservedRunningTime="2026-09-16 12:02:14.243502556 +0000 UTC m=+21.655519449"
Sep 16 12:02:14.265230 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.265175    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2s4xk" podStartSLOduration=3.87507029 podStartE2EDuration="21.265154162s" totalImagesPullingTime="17.390083872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.26494887 +0000 UTC m=+21.676965763" watchObservedRunningTime="2026-09-16 12:02:14.265154162 +0000 UTC m=+21.677171060"
Sep 16 12:02:14.285665 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.285620    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zwv6n" podStartSLOduration=3.790394127 podStartE2EDuration="21.285603184s" totalImagesPullingTime="17.495209057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.284902794 +0000 UTC m=+21.696919702" watchObservedRunningTime="2026-09-16 12:02:14.285603184 +0000 UTC m=+21.697620079"
Sep 16 12:02:14.300009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.299932    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h24wk" podStartSLOduration=7.48260318 podStartE2EDuration="21.299916103s" totalImagesPullingTime="13.817312923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.299696374 +0000 UTC m=+21.711713252" watchObservedRunningTime="2026-09-16 12:02:14.299916103 +0000 UTC m=+21.711932999"
Sep 16 12:02:14.346925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:14.346870    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j749k" podStartSLOduration=3.765523527 podStartE2EDuration="21.346852284s" totalImagesPullingTime="17.581328757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:14.346394987 +0000 UTC m=+21.758411883" watchObservedRunningTime="2026-09-16 12:02:14.346852284 +0000 UTC m=+21.758869201"
Sep 16 12:02:15.105451 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:15.105377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:15.105584 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:15.105508    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:15.105584 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:15.105377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:15.105703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:15.105377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:15.105703 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:15.105656    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:15.105803 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:15.105719    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:15.244166 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:15.244131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hsrvj" event={"ID":"11230c67-bdbb-48f7-8646-7dbf4544bd02","Type":"ContainerStarted","Data":"ccd90280a7f1ae36b57781a9d75c5e6db16b919154e2d2b8ba60d4f60e340356"}
Sep 16 12:02:15.257453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:15.257420    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:02:15.262723 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:15.262669    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hsrvj" podStartSLOduration=4.767554118 podStartE2EDuration="22.262648288s" totalImagesPullingTime="17.49509417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:15.262489568 +0000 UTC m=+22.674506463" watchObservedRunningTime="2026-09-16 12:02:15.262648288 +0000 UTC m=+22.674665186"
Sep 16 12:02:16.071051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:16.070934    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:02:15.25743895Z","UUID":"fa954d6d-9a6f-47fa-942d-d77b74952e2c","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:02:16.073035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:16.073010    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:02:16.073171 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:16.073049    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:02:16.248170 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:16.248140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5" event={"ID":"93f1d2fc-894d-4056-907f-b3d3ceabcc44","Type":"ContainerStarted","Data":"910686aee180c0b958ac9f453fd5eb85d41629522ee12e7227fff7b39016ac05"}
Sep 16 12:02:17.105222 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:17.105188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:17.105402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:17.105189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:17.105402 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:17.105303    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:17.105402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:17.105314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:17.105557 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:17.105401    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:17.105557 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:17.105488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:17.253254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:17.253219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5" event={"ID":"93f1d2fc-894d-4056-907f-b3d3ceabcc44","Type":"ContainerStarted","Data":"d0f749fc3d9f02310d37cc3f1b17fa0a939aa1a324011016c595e2df3e0b2d56"}
Sep 16 12:02:17.257016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:17.256988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"1da197ab03a3268068d3bbaf5e9d3739a54ab66d9278f7206e8e81e2de8ae934"}
Sep 16 12:02:17.270924 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:17.270879    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jn6d5" podStartSLOduration=4.447697902 podStartE2EDuration="24.270861361s" totalImagesPullingTime="19.823163459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:17.270335923 +0000 UTC m=+24.682352820" watchObservedRunningTime="2026-09-16 12:02:17.270861361 +0000 UTC m=+24.682878349"
Sep 16 12:02:18.176076 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:18.176041    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:02:18.176673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:18.176645    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:02:18.259758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:18.259727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:02:18.260224 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:18.260208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h24wk"
Sep 16 12:02:19.106325 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.106071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:19.106509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.106071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:19.106509 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:19.106357    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:19.106509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.106071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:19.106509 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:19.106410    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:19.106509 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:19.106477    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:19.264623 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.264589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" event={"ID":"7e561896-785d-4507-b5e1-54a6040be27f","Type":"ContainerStarted","Data":"03979af2f28232ce70f82c7da7b3cbb30c12cc5f9f6f71cbac80d36eee45be9c"}
Sep 16 12:02:19.265176 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.264830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:02:19.266448 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.266425    2577 generic.go:356] "Generic (PLEG): container finished" podID="39d4c6d9-8b6c-4df8-87b1-0feabaff667f" containerID="dce0792b1bea74e8f181dedd240d26bd2d08995b73d4ba9957d630327678fc39" exitCode=0
Sep 16 12:02:19.266554 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.266503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerDied","Data":"dce0792b1bea74e8f181dedd240d26bd2d08995b73d4ba9957d630327678fc39"}
Sep 16 12:02:19.280256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.280235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:02:19.290824 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:19.290782    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh" podStartSLOduration=8.732207746 podStartE2EDuration="26.290767512s" totalImagesPullingTime="17.558559766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:19.290383909 +0000 UTC m=+26.702400806" watchObservedRunningTime="2026-09-16 12:02:19.290767512 +0000 UTC m=+26.702784407"
Sep 16 12:02:20.268578 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.268526    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:02:20.269212 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.268894    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:02:20.283026 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.283001    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:02:20.582967 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.582884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lc92b"]
Sep 16 12:02:20.583113 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.583060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:20.583208 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:20.583180    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:20.586318 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.586249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-92vxd"]
Sep 16 12:02:20.586661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.586361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:20.586661 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:20.586476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:20.587010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.586985    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnxvx"]
Sep 16 12:02:20.587116 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:20.587097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:20.587224 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:20.587200    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:21.272338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:21.272308    2577 generic.go:356] "Generic (PLEG): container finished" podID="39d4c6d9-8b6c-4df8-87b1-0feabaff667f" containerID="b908891cf8cf4059a1294c9ae762a0381a5082912f150ff37a158e69d58daaeb" exitCode=0
Sep 16 12:02:21.273053 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:21.272391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerDied","Data":"b908891cf8cf4059a1294c9ae762a0381a5082912f150ff37a158e69d58daaeb"}
Sep 16 12:02:22.105770 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:22.105738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:22.105957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:22.105784    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:22.105957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:22.105864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:22.105957 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:22.105895    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:22.106114 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:22.105990    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:22.106114 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:22.106096    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:22.277224 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:22.277036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerStarted","Data":"1cb33c360a70c6f3b939aa867a74888c9d08ba14bda44c97d83acbbc224b547f"}
Sep 16 12:02:23.282645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:23.282612    2577 generic.go:356] "Generic (PLEG): container finished" podID="39d4c6d9-8b6c-4df8-87b1-0feabaff667f" containerID="1cb33c360a70c6f3b939aa867a74888c9d08ba14bda44c97d83acbbc224b547f" exitCode=0
Sep 16 12:02:23.283140 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:23.282666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerDied","Data":"1cb33c360a70c6f3b939aa867a74888c9d08ba14bda44c97d83acbbc224b547f"}
Sep 16 12:02:24.105446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:24.105407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:24.105656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:24.105406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:24.105656 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:24.105546    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:24.105656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:24.105413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:24.105656 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:24.105637    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:24.105852 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:24.105704    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:26.105543 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:26.105511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:26.106012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:26.105511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:26.106012 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.105648    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnxvx" podUID="78b0b391-4e63-4177-a34c-c226ca5bc797"
Sep 16 12:02:26.106012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:26.105511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:26.106012 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.105741    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lc92b" podUID="bdf83e47-967c-46a9-8e56-ac2a7b147088"
Sep 16 12:02:26.106012 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.105755    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92vxd" podUID="f59f7d0f-6700-43b7-b5f5-726f22dc03ea"
Sep 16 12:02:26.754556 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:26.754517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:26.754738 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.754680    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:26.754803 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.754748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret podName:bdf83e47-967c-46a9-8e56-ac2a7b147088 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.754728792 +0000 UTC m=+66.166745671 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret") pod "global-pull-secret-syncer-lc92b" (UID: "bdf83e47-967c-46a9-8e56-ac2a7b147088") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:02:26.855550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:26.855509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:26.855732 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.855689    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:26.855786 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.855775    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs podName:78b0b391-4e63-4177-a34c-c226ca5bc797 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.855755527 +0000 UTC m=+66.267772401 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs") pod "network-metrics-daemon-lnxvx" (UID: "78b0b391-4e63-4177-a34c-c226ca5bc797") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:02:26.956399 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:26.956352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:26.956604 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.956534    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:02:26.956604 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.956562    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:02:26.956604 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.956588    2577 projected.go:196] Error preparing data for projected volume kube-api-access-trpjm for pod openshift-network-diagnostics/network-check-target-92vxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:26.956753 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:26.956646    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm podName:f59f7d0f-6700-43b7-b5f5-726f22dc03ea nodeName:}" failed. No retries permitted until 2026-09-16 12:02:58.956631476 +0000 UTC m=+66.368648350 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-trpjm" (UniqueName: "kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm") pod "network-check-target-92vxd" (UID: "f59f7d0f-6700-43b7-b5f5-726f22dc03ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:02:27.424347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.424313    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-38.ec2.internal" event="NodeReady"
Sep 16 12:02:27.424819 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.424480    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:02:27.460521 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.460480    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67cd696578-lsfwf"]
Sep 16 12:02:27.489906 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.489871    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jqslb"]
Sep 16 12:02:27.490073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.490016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.492784 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.492749    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mpvqx\""
Sep 16 12:02:27.492951 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.492749    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:02:27.493469 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.493446    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:02:27.493852 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.493831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:02:27.502385 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.502362    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:02:27.506037 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.505965    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vsqll"]
Sep 16 12:02:27.506152 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.506051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:27.508708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.508682    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:02:27.509416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.509206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:02:27.509416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.509270    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rcl6b\""
Sep 16 12:02:27.509709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.509687    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:02:27.526896 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.526855    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67cd696578-lsfwf"]
Sep 16 12:02:27.526896 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.526899    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vsqll"]
Sep 16 12:02:27.527079 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.526928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jqslb"]
Sep 16 12:02:27.527079 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.526995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.529835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.529813    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:02:27.529956 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.529851    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:02:27.529956 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.529853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8hlmq\""
Sep 16 12:02:27.561407 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561380    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-bound-sa-token\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.561583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-installation-pull-secrets\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.561583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-certificates\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.561583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.561739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-trusted-ca\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.561739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-ca-trust-extracted\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.561739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b66\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-kube-api-access-w6b66\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.561739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.561657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-image-registry-private-configuration\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.662939 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.662908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-bound-sa-token\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.662951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4rs\" (UniqueName: \"kubernetes.io/projected/832aad20-0733-43cf-9a64-e417f6d13bdb-kube-api-access-mw4rs\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.663189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.662982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-installation-pull-secrets\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/832aad20-0733-43cf-9a64-e417f6d13bdb-config-volume\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.663189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-certificates\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f193c68b-22e0-47b7-bad7-a883c871948a-cert\") pod \"ingress-canary-jqslb\" (UID: \"f193c68b-22e0-47b7-bad7-a883c871948a\") " pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-trusted-ca\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-ca-trust-extracted\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aad20-0733-43cf-9a64-e417f6d13bdb-tmp-dir\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:27.663305    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4rv\" (UniqueName: \"kubernetes.io/projected/f193c68b-22e0-47b7-bad7-a883c871948a-kube-api-access-gp4rv\") pod \"ingress-canary-jqslb\" (UID: \"f193c68b-22e0-47b7-bad7-a883c871948a\") " pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:27.663325    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cd696578-lsfwf: secret "image-registry-tls" not found
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b66\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-kube-api-access-w6b66\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:27.663393    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls podName:4a892003-41c7-4f2a-9b29-3c9fad4b0f83 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:28.163371901 +0000 UTC m=+35.575388796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls") pod "image-registry-67cd696578-lsfwf" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83") : secret "image-registry-tls" not found
Sep 16 12:02:27.663497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-image-registry-private-configuration\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663976 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-ca-trust-extracted\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.663976 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.663839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-certificates\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.664382 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.664360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-trusted-ca\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.667600 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.667532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-installation-pull-secrets\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.667600 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.667543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-image-registry-private-configuration\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.677269 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.677191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-bound-sa-token\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.677269 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.677242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b66\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-kube-api-access-w6b66\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:27.764308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4rs\" (UniqueName: \"kubernetes.io/projected/832aad20-0733-43cf-9a64-e417f6d13bdb-kube-api-access-mw4rs\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.764308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/832aad20-0733-43cf-9a64-e417f6d13bdb-config-volume\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.764509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f193c68b-22e0-47b7-bad7-a883c871948a-cert\") pod \"ingress-canary-jqslb\" (UID: \"f193c68b-22e0-47b7-bad7-a883c871948a\") " pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:27.764509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.764509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aad20-0733-43cf-9a64-e417f6d13bdb-tmp-dir\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.764509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4rv\" (UniqueName: \"kubernetes.io/projected/f193c68b-22e0-47b7-bad7-a883c871948a-kube-api-access-gp4rv\") pod \"ingress-canary-jqslb\" (UID: \"f193c68b-22e0-47b7-bad7-a883c871948a\") " pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:27.764728 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:27.764619    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:27.764728 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:27.764707    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls podName:832aad20-0733-43cf-9a64-e417f6d13bdb nodeName:}" failed. No retries permitted until 2026-09-16 12:02:28.264684586 +0000 UTC m=+35.676701484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls") pod "dns-default-vsqll" (UID: "832aad20-0733-43cf-9a64-e417f6d13bdb") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:27.764837 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aad20-0733-43cf-9a64-e417f6d13bdb-tmp-dir\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.765000 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.764979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/832aad20-0733-43cf-9a64-e417f6d13bdb-config-volume\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.766764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.766734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f193c68b-22e0-47b7-bad7-a883c871948a-cert\") pod \"ingress-canary-jqslb\" (UID: \"f193c68b-22e0-47b7-bad7-a883c871948a\") " pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:27.772821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.772802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4rs\" (UniqueName: \"kubernetes.io/projected/832aad20-0733-43cf-9a64-e417f6d13bdb-kube-api-access-mw4rs\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:27.773363 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.773342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4rv\" (UniqueName: \"kubernetes.io/projected/f193c68b-22e0-47b7-bad7-a883c871948a-kube-api-access-gp4rv\") pod \"ingress-canary-jqslb\" (UID: \"f193c68b-22e0-47b7-bad7-a883c871948a\") " pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:27.817441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:27.817408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jqslb"
Sep 16 12:02:28.105483 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.105247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:28.105611 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.105265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:28.105775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.105284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:28.108083 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.108067    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:02:28.108209 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.108192    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:02:28.108352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.108331    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:02:28.108632 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.108615    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:02:28.109245 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.109226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qzcvp\""
Sep 16 12:02:28.109350 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.109252    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pcfgd\""
Sep 16 12:02:28.117366 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.117340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jqslb"]
Sep 16 12:02:28.121347 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:28.121099    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf193c68b_22e0_47b7_bad7_a883c871948a.slice/crio-bd93873b63209dbf7ed44076220658e66e6d1b1f487f4cf073c2c26fa60358e1 WatchSource:0}: Error finding container bd93873b63209dbf7ed44076220658e66e6d1b1f487f4cf073c2c26fa60358e1: Status 404 returned error can't find the container with id bd93873b63209dbf7ed44076220658e66e6d1b1f487f4cf073c2c26fa60358e1
Sep 16 12:02:28.167496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.167472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:28.167685 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:28.167666    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:28.167749 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:28.167690    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cd696578-lsfwf: secret "image-registry-tls" not found
Sep 16 12:02:28.167784 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:28.167771    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls podName:4a892003-41c7-4f2a-9b29-3c9fad4b0f83 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:29.167746239 +0000 UTC m=+36.579763126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls") pod "image-registry-67cd696578-lsfwf" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83") : secret "image-registry-tls" not found
Sep 16 12:02:28.268301 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.268263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:28.268446 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:28.268382    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:28.268446 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:28.268435    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls podName:832aad20-0733-43cf-9a64-e417f6d13bdb nodeName:}" failed. No retries permitted until 2026-09-16 12:02:29.268420938 +0000 UTC m=+36.680437824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls") pod "dns-default-vsqll" (UID: "832aad20-0733-43cf-9a64-e417f6d13bdb") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:28.296315 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.296282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jqslb" event={"ID":"f193c68b-22e0-47b7-bad7-a883c871948a","Type":"ContainerStarted","Data":"bd93873b63209dbf7ed44076220658e66e6d1b1f487f4cf073c2c26fa60358e1"}
Sep 16 12:02:28.299339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:28.299313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerStarted","Data":"35e1d8a6b6e2832c514017d1235f8ea72e3fd047cfa2a3461f1c4cd9dccf195c"}
Sep 16 12:02:29.176211 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:29.176177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:29.176886 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:29.176345    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:29.176886 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:29.176364    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cd696578-lsfwf: secret "image-registry-tls" not found
Sep 16 12:02:29.176886 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:29.176426    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls podName:4a892003-41c7-4f2a-9b29-3c9fad4b0f83 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:31.176404788 +0000 UTC m=+38.588421665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls") pod "image-registry-67cd696578-lsfwf" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83") : secret "image-registry-tls" not found
Sep 16 12:02:29.277259 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:29.277215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:29.277423 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:29.277391    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:29.277489 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:29.277471    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls podName:832aad20-0733-43cf-9a64-e417f6d13bdb nodeName:}" failed. No retries permitted until 2026-09-16 12:02:31.277449842 +0000 UTC m=+38.689466737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls") pod "dns-default-vsqll" (UID: "832aad20-0733-43cf-9a64-e417f6d13bdb") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:29.305137 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:29.305103    2577 generic.go:356] "Generic (PLEG): container finished" podID="39d4c6d9-8b6c-4df8-87b1-0feabaff667f" containerID="35e1d8a6b6e2832c514017d1235f8ea72e3fd047cfa2a3461f1c4cd9dccf195c" exitCode=0
Sep 16 12:02:29.305298 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:29.305158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerDied","Data":"35e1d8a6b6e2832c514017d1235f8ea72e3fd047cfa2a3461f1c4cd9dccf195c"}
Sep 16 12:02:30.311352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:30.311309    2577 generic.go:356] "Generic (PLEG): container finished" podID="39d4c6d9-8b6c-4df8-87b1-0feabaff667f" containerID="e91c6f00f47ee9f9bd3738e07f1ee2dd73777c9cbed6548055f623f4e8a36880" exitCode=0
Sep 16 12:02:30.311779 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:30.311373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerDied","Data":"e91c6f00f47ee9f9bd3738e07f1ee2dd73777c9cbed6548055f623f4e8a36880"}
Sep 16 12:02:31.192365 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.192320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:31.192503 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:31.192471    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:31.192503 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:31.192492    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cd696578-lsfwf: secret "image-registry-tls" not found
Sep 16 12:02:31.192615 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:31.192553    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls podName:4a892003-41c7-4f2a-9b29-3c9fad4b0f83 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:35.192535177 +0000 UTC m=+42.604552066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls") pod "image-registry-67cd696578-lsfwf" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83") : secret "image-registry-tls" not found
Sep 16 12:02:31.274407 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.274324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:31.293429 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.293392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:31.293589 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:31.293548    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:31.293642 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:31.293635    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls podName:832aad20-0733-43cf-9a64-e417f6d13bdb nodeName:}" failed. No retries permitted until 2026-09-16 12:02:35.293617143 +0000 UTC m=+42.705634017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls") pod "dns-default-vsqll" (UID: "832aad20-0733-43cf-9a64-e417f6d13bdb") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:31.316889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.316852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" event={"ID":"39d4c6d9-8b6c-4df8-87b1-0feabaff667f","Type":"ContainerStarted","Data":"70e3bc384dec6ba8b40a7b6f8148c7b032a447e2f8192d4fd4ea328d48a78912"}
Sep 16 12:02:31.343491 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.343441    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k8r7h" podStartSLOduration=9.175560932 podStartE2EDuration="38.343424396s" totalImagesPullingTime="29.167863464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:31.342769946 +0000 UTC m=+38.754786842" watchObservedRunningTime="2026-09-16 12:02:31.343424396 +0000 UTC m=+38.755441291"
Sep 16 12:02:31.360984 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.360959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:31.461986 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.461954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:31.562660 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.562559    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:31.621586 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.621398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:31.752293 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:31.752254    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:32.061098 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.061063    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"]
Sep 16 12:02:32.063821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.063801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:32.066155 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.066133    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 12:02:32.066274 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.066185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tqkcc\""
Sep 16 12:02:32.067391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.067367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 12:02:32.071320 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.071301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"]
Sep 16 12:02:32.186913 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.186880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:32.200012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.199977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a467175a-cb94-403f-a278-ce1af1396e85-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:32.200075 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.200024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:32.253886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.253855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:32.300637 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.300606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a467175a-cb94-403f-a278-ce1af1396e85-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:32.300738 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.300652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:32.300795 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:32.300783    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:02:32.300849 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:32.300840    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert podName:a467175a-cb94-403f-a278-ce1af1396e85 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:32.800825237 +0000 UTC m=+40.212842111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c4qlf" (UID: "a467175a-cb94-403f-a278-ce1af1396e85") : secret "networking-console-plugin-cert" not found
Sep 16 12:02:32.301254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.301234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a467175a-cb94-403f-a278-ce1af1396e85-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:32.320931 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.320849    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jqslb" event={"ID":"f193c68b-22e0-47b7-bad7-a883c871948a","Type":"ContainerStarted","Data":"d9c7312b83333bad2a2e7a807137ab7f3eaf6cc1740893d9d5899e82d0bf5c02"}
Sep 16 12:02:32.340757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.340705    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jqslb" podStartSLOduration=2.114150438 podStartE2EDuration="5.340689082s" totalImagesPullingTime="3.226538644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:32.340276341 +0000 UTC m=+39.752293237" watchObservedRunningTime="2026-09-16 12:02:32.340689082 +0000 UTC m=+39.752705969"
Sep 16 12:02:32.612314 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.612233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:32.804417 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:32.804377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:32.804587 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:32.804488    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:02:32.804587 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:32.804540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert podName:a467175a-cb94-403f-a278-ce1af1396e85 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:33.804526789 +0000 UTC m=+41.216543676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c4qlf" (UID: "a467175a-cb94-403f-a278-ce1af1396e85") : secret "networking-console-plugin-cert" not found
Sep 16 12:02:33.004999 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:33.004950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:33.117822 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:33.117792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:33.811059 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:33.811024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:33.811416 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:33.811136    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:02:33.811416 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:33.811192    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert podName:a467175a-cb94-403f-a278-ce1af1396e85 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:35.811179063 +0000 UTC m=+43.223195936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c4qlf" (UID: "a467175a-cb94-403f-a278-ce1af1396e85") : secret "networking-console-plugin-cert" not found
Sep 16 12:02:35.222235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:35.222201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:35.222661 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:35.222338    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:35.222661 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:35.222351    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cd696578-lsfwf: secret "image-registry-tls" not found
Sep 16 12:02:35.222661 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:35.222405    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls podName:4a892003-41c7-4f2a-9b29-3c9fad4b0f83 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:43.222388003 +0000 UTC m=+50.634404905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls") pod "image-registry-67cd696578-lsfwf" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83") : secret "image-registry-tls" not found
Sep 16 12:02:35.322559 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:35.322527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:35.322729 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:35.322704    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:35.322804 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:35.322791    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls podName:832aad20-0733-43cf-9a64-e417f6d13bdb nodeName:}" failed. No retries permitted until 2026-09-16 12:02:43.322768952 +0000 UTC m=+50.734785846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls") pod "dns-default-vsqll" (UID: "832aad20-0733-43cf-9a64-e417f6d13bdb") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:35.825496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:35.825451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:35.825701 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:35.825625    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:02:35.825701 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:35.825688    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert podName:a467175a-cb94-403f-a278-ce1af1396e85 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:39.825673462 +0000 UTC m=+47.237690347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c4qlf" (UID: "a467175a-cb94-403f-a278-ce1af1396e85") : secret "networking-console-plugin-cert" not found
Sep 16 12:02:36.289195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:36.289157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:36.392025 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:36.391989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:36.491862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:36.491830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:36.750017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:36.749985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:36.813580 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:36.813533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:37.493106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.493080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:37.498554 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.498535    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-22d4w"]
Sep 16 12:02:37.541525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.541503    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22d4w"]
Sep 16 12:02:37.541657 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.541632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.544335 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.544308    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:02:37.544581 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.544553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:02:37.544677 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.544557    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4nl5\""
Sep 16 12:02:37.545504 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.545480    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:02:37.545578 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.545519    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:02:37.563243 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.563219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:37.640508 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.640480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.640647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.640513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.640647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.640531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhq9\" (UniqueName: \"kubernetes.io/projected/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-api-access-pnhq9\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.640647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.640618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df95d8a0-b180-405e-ac4b-7b1d44a792db-crio-socket\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.640748 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.640674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df95d8a0-b180-405e-ac4b-7b1d44a792db-data-volume\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.655869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.655842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:37.741787 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.741756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.741899 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.741793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.741899 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.741820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhq9\" (UniqueName: \"kubernetes.io/projected/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-api-access-pnhq9\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.741899 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.741877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df95d8a0-b180-405e-ac4b-7b1d44a792db-crio-socket\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.741899 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:37.741891    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:37.742083 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:37.741952    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:38.241937945 +0000 UTC m=+45.653954835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:37.742083 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:37.741893    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:37.742083 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.741982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df95d8a0-b180-405e-ac4b-7b1d44a792db-data-volume\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.742083 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:37.742035    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:38.242016642 +0000 UTC m=+45.654033520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:37.742242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.742109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/df95d8a0-b180-405e-ac4b-7b1d44a792db-crio-socket\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.742281 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.742238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/df95d8a0-b180-405e-ac4b-7b1d44a792db-data-volume\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.751304 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.751255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhq9\" (UniqueName: \"kubernetes.io/projected/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-api-access-pnhq9\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:37.756225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:37.756202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.171096 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.171063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.245985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.245956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:38.245985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.245989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:38.246156 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:38.246073    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:38.246156 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:38.246091    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:38.246156 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:38.246141    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:39.24612431 +0000 UTC m=+46.658141204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:38.246156 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:38.246155    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:39.246149005 +0000 UTC m=+46.658165878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:38.272229 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.272204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.377222 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.377191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.472363 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.472292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.575309 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.575276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.678763 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.678736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.801593 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.801501    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2s4xk_4942bb0d-de9b-4af1-8ece-2d67d2a91f51/dns-node-resolver/0.log"
Sep 16 12:02:38.887261 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.887231    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:38.992564 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:38.992523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:39.253492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.253462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:39.253661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.253499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:39.253661 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:39.253597    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:39.253661 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:39.253627    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:39.253661 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:39.253653    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:41.253639212 +0000 UTC m=+48.665656090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:39.253802 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:39.253677    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:41.253665154 +0000 UTC m=+48.665682051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:39.349338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.349309    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"]
Sep 16 12:02:39.369900 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.369878    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"]
Sep 16 12:02:39.370009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.369966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"
Sep 16 12:02:39.372414 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.372385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 12:02:39.372520 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.372386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 12:02:39.372520 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.372447    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t7w45\""
Sep 16 12:02:39.388963 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.388944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:39.456068 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.456045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjp8\" (UniqueName: \"kubernetes.io/projected/7d7ea7b5-9015-4371-a197-89f361dd6e92-kube-api-access-fqjp8\") pod \"migrator-8b4d697c5-xtx9c\" (UID: \"7d7ea7b5-9015-4371-a197-89f361dd6e92\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"
Sep 16 12:02:39.487508 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.487484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:39.556476 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.556416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjp8\" (UniqueName: \"kubernetes.io/projected/7d7ea7b5-9015-4371-a197-89f361dd6e92-kube-api-access-fqjp8\") pod \"migrator-8b4d697c5-xtx9c\" (UID: \"7d7ea7b5-9015-4371-a197-89f361dd6e92\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"
Sep 16 12:02:39.564058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.564040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjp8\" (UniqueName: \"kubernetes.io/projected/7d7ea7b5-9015-4371-a197-89f361dd6e92-kube-api-access-fqjp8\") pod \"migrator-8b4d697c5-xtx9c\" (UID: \"7d7ea7b5-9015-4371-a197-89f361dd6e92\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"
Sep 16 12:02:39.590390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.590365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:39.678527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.678501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"
Sep 16 12:02:39.858854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.858777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:39.858983 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:39.858929    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:02:39.859020 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:39.859008    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert podName:a467175a-cb94-403f-a278-ce1af1396e85 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:47.85899199 +0000 UTC m=+55.271008878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c4qlf" (UID: "a467175a-cb94-403f-a278-ce1af1396e85") : secret "networking-console-plugin-cert" not found
Sep 16 12:02:39.960895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:39.960876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c"]
Sep 16 12:02:40.000485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.000466    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cvcd6_db0b38f7-df0b-4486-9151-97c4fe311900/node-ca/0.log"
Sep 16 12:02:40.339841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.339810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c" event={"ID":"7d7ea7b5-9015-4371-a197-89f361dd6e92","Type":"ContainerStarted","Data":"188f70a04f78ef46b6f339644357d4b5836e3dcb3469e12fd6f9d74f359efcbe"}
Sep 16 12:02:40.424832 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.424799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:40.477656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.477629    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6phdt"]
Sep 16 12:02:40.508408 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.508387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6phdt"]
Sep 16 12:02:40.508536 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.508468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.511120 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.511102    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 12:02:40.512232 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.512213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 12:02:40.512232 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.512226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g7rbr\""
Sep 16 12:02:40.512374 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.512213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 12:02:40.512374 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.512214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 12:02:40.525110 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.525091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:40.584465 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.584440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:40.600670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.600605    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jqslb_f193c68b-22e0-47b7-bad7-a883c871948a/serve-healthcheck-canary/0.log"
Sep 16 12:02:40.663873 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.663845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5283cde-26cf-4e42-a370-592fc00d4452-signing-cabundle\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.664054 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.664023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5283cde-26cf-4e42-a370-592fc00d4452-signing-key\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.664175 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.664069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprwm\" (UniqueName: \"kubernetes.io/projected/b5283cde-26cf-4e42-a370-592fc00d4452-kube-api-access-cprwm\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.764757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.764689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprwm\" (UniqueName: \"kubernetes.io/projected/b5283cde-26cf-4e42-a370-592fc00d4452-kube-api-access-cprwm\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.764905 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.764851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5283cde-26cf-4e42-a370-592fc00d4452-signing-cabundle\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.765109 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.765022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5283cde-26cf-4e42-a370-592fc00d4452-signing-key\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.768140 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.765602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5283cde-26cf-4e42-a370-592fc00d4452-signing-cabundle\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.769053 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.769003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5283cde-26cf-4e42-a370-592fc00d4452-signing-key\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.774256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.774237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprwm\" (UniqueName: \"kubernetes.io/projected/b5283cde-26cf-4e42-a370-592fc00d4452-kube-api-access-cprwm\") pod \"service-ca-568869fdc5-6phdt\" (UID: \"b5283cde-26cf-4e42-a370-592fc00d4452\") " pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:40.817216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:40.817186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6phdt"
Sep 16 12:02:41.109595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:41.109553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6phdt"]
Sep 16 12:02:41.237807 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:41.237720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:41.269634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:41.269603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:41.269750 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:41.269641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:41.269750 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:41.269710    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:02:41.269750 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:41.269739    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:41.269858 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:41.269758    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:45.269744251 +0000 UTC m=+52.681761126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : configmap "kube-rbac-proxy" not found
Sep 16 12:02:41.269858 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:41.269799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:45.269785179 +0000 UTC m=+52.681802071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:41.321467 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:41.321440    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5283cde_26cf_4e42_a370_592fc00d4452.slice/crio-413cb33f122771ad7b91fe0679cf8cd92fa196a9a335506bfd30a56577e21410 WatchSource:0}: Error finding container 413cb33f122771ad7b91fe0679cf8cd92fa196a9a335506bfd30a56577e21410: Status 404 returned error can't find the container with id 413cb33f122771ad7b91fe0679cf8cd92fa196a9a335506bfd30a56577e21410
Sep 16 12:02:41.342558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:41.342535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6phdt" event={"ID":"b5283cde-26cf-4e42-a370-592fc00d4452","Type":"ContainerStarted","Data":"413cb33f122771ad7b91fe0679cf8cd92fa196a9a335506bfd30a56577e21410"}
Sep 16 12:02:42.348286 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:42.348239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c" event={"ID":"7d7ea7b5-9015-4371-a197-89f361dd6e92","Type":"ContainerStarted","Data":"200b3e081387fe245f300c1b076c780a22b85e4e0b8ed111b3fb1b4d3c3ff0ee"}
Sep 16 12:02:42.348746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:42.348303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c" event={"ID":"7d7ea7b5-9015-4371-a197-89f361dd6e92","Type":"ContainerStarted","Data":"899f3103194cb48722a3ed1644df9db7425f88a03f9b107081049b2285647f02"}
Sep 16 12:02:42.365886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:42.365827    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-xtx9c" podStartSLOduration=1.6466467169999999 podStartE2EDuration="3.365811085s" totalImagesPullingTime="1.719164368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:42.36489547 +0000 UTC m=+49.776912367" watchObservedRunningTime="2026-09-16 12:02:42.365811085 +0000 UTC m=+49.777827980"
Sep 16 12:02:42.734038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:42.734008    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:43.285512 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:43.285489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:43.285675 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:43.285654    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:02:43.285745 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:43.285679    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cd696578-lsfwf: secret "image-registry-tls" not found
Sep 16 12:02:43.285802 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:43.285744    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls podName:4a892003-41c7-4f2a-9b29-3c9fad4b0f83 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:59.285722942 +0000 UTC m=+66.697739830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls") pod "image-registry-67cd696578-lsfwf" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83") : secret "image-registry-tls" not found
Sep 16 12:02:43.352420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:43.352384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6phdt" event={"ID":"b5283cde-26cf-4e42-a370-592fc00d4452","Type":"ContainerStarted","Data":"c4c2a9de2512c0ce26a38bea906879f2e5d20785d245930e8b73f132fbf2c196"}
Sep 16 12:02:43.368671 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:43.368622    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-6phdt" podStartSLOduration=1.442775589 podStartE2EDuration="3.368606538s" totalImagesPullingTime="1.925830949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:43.367836227 +0000 UTC m=+50.779853125" watchObservedRunningTime="2026-09-16 12:02:43.368606538 +0000 UTC m=+50.780623433"
Sep 16 12:02:43.386104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:43.386075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:43.386249 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:43.386232    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:02:43.386311 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:43.386298    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls podName:832aad20-0733-43cf-9a64-e417f6d13bdb nodeName:}" failed. No retries permitted until 2026-09-16 12:02:59.386280411 +0000 UTC m=+66.798297304 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls") pod "dns-default-vsqll" (UID: "832aad20-0733-43cf-9a64-e417f6d13bdb") : secret "dns-default-metrics-tls" not found
Sep 16 12:02:44.549483 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:44.549436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:44.726707 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:44.726674    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:44.826524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:44.826424    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:44.891472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:44.891429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:44.994554 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:44.994516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:45.102217 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:45.102127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:45.203485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:45.203456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:45.297097 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:45.297062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:45.300685 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:45.300666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:45.300755 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:45.300693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:45.300836 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:45.300820    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:45.300903 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:45.300893    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls podName:df95d8a0-b180-405e-ac4b-7b1d44a792db nodeName:}" failed. No retries permitted until 2026-09-16 12:02:53.300873116 +0000 UTC m=+60.712890005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22d4w" (UID: "df95d8a0-b180-405e-ac4b-7b1d44a792db") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:02:45.301130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:45.301114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/df95d8a0-b180-405e-ac4b-7b1d44a792db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:46.047434 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:46.047400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:46.448661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:46.448626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:46.855794 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:46.855706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:47.293959 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:47.293928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:47.453754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:47.453716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:47.653150 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:47.653114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:47.922371 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:47.922287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:02:47.922503 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:47.922406    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:02:47.922503 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:02:47.922463    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert podName:a467175a-cb94-403f-a278-ce1af1396e85 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:03.922447177 +0000 UTC m=+71.334464053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-c4qlf" (UID: "a467175a-cb94-403f-a278-ce1af1396e85") : secret "networking-console-plugin-cert" not found
Sep 16 12:02:48.251145 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:48.251068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:48.434205 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:48.434167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:48.856342 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:48.856310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:49.055433 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:49.055404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:49.250504 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:49.250473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:49.309147 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:49.309117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:49.452893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:49.452861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:49.646555 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:49.646528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:50.667357 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:50.667320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:50.768250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:50.768219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:51.052419 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:51.052337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:51.254263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:51.254233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:51.453776 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:51.453746    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:51.794068 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:51.793993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:52.286583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:52.286540    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xtlh"
Sep 16 12:02:52.850757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:52.850724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:52.910809 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:52.910777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:53.053449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.053416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:53.252356 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.252319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:53.362355 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.362323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:53.364684 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.364657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/df95d8a0-b180-405e-ac4b-7b1d44a792db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22d4w\" (UID: \"df95d8a0-b180-405e-ac4b-7b1d44a792db\") " pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:53.454011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.453982    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4nl5\""
Sep 16 12:02:53.456692 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.456669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:53.462549 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.462532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22d4w"
Sep 16 12:02:53.658245 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.658214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:53.731292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:53.731256    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22d4w"]
Sep 16 12:02:53.733970 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:53.733941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf95d8a0_b180_405e_ac4b_7b1d44a792db.slice/crio-73e46274aa3a8e511022fdfbaad8c63f00a9e5a9d830ba8fdf6ae516526f73a8 WatchSource:0}: Error finding container 73e46274aa3a8e511022fdfbaad8c63f00a9e5a9d830ba8fdf6ae516526f73a8: Status 404 returned error can't find the container with id 73e46274aa3a8e511022fdfbaad8c63f00a9e5a9d830ba8fdf6ae516526f73a8
Sep 16 12:02:54.380456 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:54.380422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22d4w" event={"ID":"df95d8a0-b180-405e-ac4b-7b1d44a792db","Type":"ContainerStarted","Data":"a8da14f83dd9cf5582074eef595ca8731b9df62c503d5b7de225c21829fe520b"}
Sep 16 12:02:54.380829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:54.380500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22d4w" event={"ID":"df95d8a0-b180-405e-ac4b-7b1d44a792db","Type":"ContainerStarted","Data":"73e46274aa3a8e511022fdfbaad8c63f00a9e5a9d830ba8fdf6ae516526f73a8"}
Sep 16 12:02:54.459635 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:54.459606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:54.846423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:54.846351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:55.022723 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:55.022686    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:55.122440 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:55.122353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:55.274822 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:55.274787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:55.386392 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:55.386358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22d4w" event={"ID":"df95d8a0-b180-405e-ac4b-7b1d44a792db","Type":"ContainerStarted","Data":"d243a19a4738ab39b5d41ca063e149568daf4307042255dafc611996cf160260"}
Sep 16 12:02:55.426997 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:55.426967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:55.527842 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:55.527805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:02:56.393994 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:56.393962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22d4w" event={"ID":"df95d8a0-b180-405e-ac4b-7b1d44a792db","Type":"ContainerStarted","Data":"4db75204e26bcbe5f832cc3dcfb97947e3ce601e67f8d416fc2b784ec508a5ff"}
Sep 16 12:02:56.412464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:56.412421    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-22d4w" podStartSLOduration=17.411620975 podStartE2EDuration="19.412407582s" totalImagesPullingTime="2.000786607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:02:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:56.410941123 +0000 UTC m=+63.822958018" watchObservedRunningTime="2026-09-16 12:02:56.412407582 +0000 UTC m=+63.824424478"
Sep 16 12:02:58.802317 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:58.802284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:58.805051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:58.805032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:02:58.815525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:58.815494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf83e47-967c-46a9-8e56-ac2a7b147088-original-pull-secret\") pod \"global-pull-secret-syncer-lc92b\" (UID: \"bdf83e47-967c-46a9-8e56-ac2a7b147088\") " pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:58.903516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:58.903481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:58.906489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:58.906470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:02:58.916014 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:58.915986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b0b391-4e63-4177-a34c-c226ca5bc797-metrics-certs\") pod \"network-metrics-daemon-lnxvx\" (UID: \"78b0b391-4e63-4177-a34c-c226ca5bc797\") " pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:59.004804 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.004753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:59.008398 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.008379    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:02:59.018445 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.018424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:02:59.027075 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.027050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qzcvp\""
Sep 16 12:02:59.028952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.028931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trpjm\" (UniqueName: \"kubernetes.io/projected/f59f7d0f-6700-43b7-b5f5-726f22dc03ea-kube-api-access-trpjm\") pod \"network-check-target-92vxd\" (UID: \"f59f7d0f-6700-43b7-b5f5-726f22dc03ea\") " pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:59.031492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.031465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lc92b"
Sep 16 12:02:59.035122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.035108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnxvx"
Sep 16 12:02:59.303369 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.303344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lc92b"]
Sep 16 12:02:59.305711 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:59.305674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf83e47_967c_46a9_8e56_ac2a7b147088.slice/crio-25e184bf14608d47acbd0ef50d1c3995977bdd53fd5e33b34b2ce69ebd2df38e WatchSource:0}: Error finding container 25e184bf14608d47acbd0ef50d1c3995977bdd53fd5e33b34b2ce69ebd2df38e: Status 404 returned error can't find the container with id 25e184bf14608d47acbd0ef50d1c3995977bdd53fd5e33b34b2ce69ebd2df38e
Sep 16 12:02:59.306848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.306818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:59.306848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.306835    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnxvx"]
Sep 16 12:02:59.309339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.309317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"image-registry-67cd696578-lsfwf\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") " pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:59.310738 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:59.310712    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b0b391_4e63_4177_a34c_c226ca5bc797.slice/crio-13bdc01c4d1a17f23bf01ae0799936eea6ef3d2b9cb9d073368a0301308dd1d4 WatchSource:0}: Error finding container 13bdc01c4d1a17f23bf01ae0799936eea6ef3d2b9cb9d073368a0301308dd1d4: Status 404 returned error can't find the container with id 13bdc01c4d1a17f23bf01ae0799936eea6ef3d2b9cb9d073368a0301308dd1d4
Sep 16 12:02:59.319977 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.319960    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pcfgd\""
Sep 16 12:02:59.328144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.328124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:02:59.404281 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.404251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lc92b" event={"ID":"bdf83e47-967c-46a9-8e56-ac2a7b147088","Type":"ContainerStarted","Data":"25e184bf14608d47acbd0ef50d1c3995977bdd53fd5e33b34b2ce69ebd2df38e"}
Sep 16 12:02:59.405415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.405390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnxvx" event={"ID":"78b0b391-4e63-4177-a34c-c226ca5bc797","Type":"ContainerStarted","Data":"13bdc01c4d1a17f23bf01ae0799936eea6ef3d2b9cb9d073368a0301308dd1d4"}
Sep 16 12:02:59.407790 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.407773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:59.409881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.409859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/832aad20-0733-43cf-9a64-e417f6d13bdb-metrics-tls\") pod \"dns-default-vsqll\" (UID: \"832aad20-0733-43cf-9a64-e417f6d13bdb\") " pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:59.592412 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.592334    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-92vxd"]
Sep 16 12:02:59.594877 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:59.594854    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59f7d0f_6700_43b7_b5f5_726f22dc03ea.slice/crio-6a7a252db55750a677841bde4e59127b9c55b0c57a06882423807b98105e1478 WatchSource:0}: Error finding container 6a7a252db55750a677841bde4e59127b9c55b0c57a06882423807b98105e1478: Status 404 returned error can't find the container with id 6a7a252db55750a677841bde4e59127b9c55b0c57a06882423807b98105e1478
Sep 16 12:02:59.605114 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.605096    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mpvqx\""
Sep 16 12:02:59.613424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.613405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:02:59.639888 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.639863    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8hlmq\""
Sep 16 12:02:59.647829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.647771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vsqll"
Sep 16 12:02:59.887292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.887218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67cd696578-lsfwf"]
Sep 16 12:02:59.891207 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:59.891178    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a892003_41c7_4f2a_9b29_3c9fad4b0f83.slice/crio-3dbdeea520d5e25f83ddbba04d91bb8852a956ff19b4baa9a141ccec9bfce2d9 WatchSource:0}: Error finding container 3dbdeea520d5e25f83ddbba04d91bb8852a956ff19b4baa9a141ccec9bfce2d9: Status 404 returned error can't find the container with id 3dbdeea520d5e25f83ddbba04d91bb8852a956ff19b4baa9a141ccec9bfce2d9
Sep 16 12:02:59.928785 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:02:59.928764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vsqll"]
Sep 16 12:02:59.931450 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:02:59.931424    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832aad20_0733_43cf_9a64_e417f6d13bdb.slice/crio-2d25f609f0e51a8f09a0d7ed7ceb499f6ebe7a174ab7d86576c278826826befe WatchSource:0}: Error finding container 2d25f609f0e51a8f09a0d7ed7ceb499f6ebe7a174ab7d86576c278826826befe: Status 404 returned error can't find the container with id 2d25f609f0e51a8f09a0d7ed7ceb499f6ebe7a174ab7d86576c278826826befe
Sep 16 12:03:00.076863 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.076826    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"]
Sep 16 12:03:00.079316 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.079186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.082760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.082732    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"]
Sep 16 12:03:00.085246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.084785    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67cd696578-lsfwf"]
Sep 16 12:03:00.085246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.084890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.088645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.088184    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:03:00.088645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.088192    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 12:03:00.088645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.088462    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 12:03:00.088645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.088543    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:03:00.089521 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.089430    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:03:00.089644 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.089524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 12:03:00.089710 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.089685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-t8b2z\""
Sep 16 12:03:00.090076 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.089955    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 12:03:00.090195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.090177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 12:03:00.097753 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.097728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"]
Sep 16 12:03:00.104618 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.104591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"]
Sep 16 12:03:00.114290 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-ca\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.114419 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-hub\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.114419 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b102341d-cead-4a14-880e-3c6f5657fbfb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4\" (UID: \"b102341d-cead-4a14-880e-3c6f5657fbfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.114419 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbtq\" (UniqueName: \"kubernetes.io/projected/b102341d-cead-4a14-880e-3c6f5657fbfb-kube-api-access-msbtq\") pod \"managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4\" (UID: \"b102341d-cead-4a14-880e-3c6f5657fbfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.114608 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.114608 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfrjh\" (UniqueName: \"kubernetes.io/projected/48e1307d-e38f-4713-bf53-891fb47b8094-kube-api-access-tfrjh\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.114608 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.114608 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.114528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e1307d-e38f-4713-bf53-891fb47b8094-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.204641 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.204104    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6958b676fb-fvx7b"]
Sep 16 12:03:00.206639 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.206609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.215339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.215533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfrjh\" (UniqueName: \"kubernetes.io/projected/48e1307d-e38f-4713-bf53-891fb47b8094-kube-api-access-tfrjh\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.215533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.215533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e1307d-e38f-4713-bf53-891fb47b8094-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.215533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-ca\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.215533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-hub\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.215533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b102341d-cead-4a14-880e-3c6f5657fbfb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4\" (UID: \"b102341d-cead-4a14-880e-3c6f5657fbfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.215866 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.215542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbtq\" (UniqueName: \"kubernetes.io/projected/b102341d-cead-4a14-880e-3c6f5657fbfb-kube-api-access-msbtq\") pod \"managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4\" (UID: \"b102341d-cead-4a14-880e-3c6f5657fbfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.216398 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.216370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e1307d-e38f-4713-bf53-891fb47b8094-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.218944 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.218919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-ca\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.219045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.218948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b102341d-cead-4a14-880e-3c6f5657fbfb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4\" (UID: \"b102341d-cead-4a14-880e-3c6f5657fbfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.219402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.219363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.219490 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.219455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-hub\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.219671 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.219625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48e1307d-e38f-4713-bf53-891fb47b8094-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.244463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.244431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbtq\" (UniqueName: \"kubernetes.io/projected/b102341d-cead-4a14-880e-3c6f5657fbfb-kube-api-access-msbtq\") pod \"managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4\" (UID: \"b102341d-cead-4a14-880e-3c6f5657fbfb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.247132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.247068    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6958b676fb-fvx7b"]
Sep 16 12:03:00.255364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.255326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfrjh\" (UniqueName: \"kubernetes.io/projected/48e1307d-e38f-4713-bf53-891fb47b8094-kube-api-access-tfrjh\") pod \"cluster-proxy-proxy-agent-5b64d9cb46-7cwkx\" (UID: \"48e1307d-e38f-4713-bf53-891fb47b8094\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.285425 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.285384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:00.317480 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-registry-certificates\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.317666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-installation-pull-secrets\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.317666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-registry-tls\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.317666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-trusted-ca\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.317666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bnk\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-kube-api-access-g9bnk\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.317884 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-ca-trust-extracted\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.317884 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-bound-sa-token\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.317884 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.317798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-image-registry-private-configuration\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.394525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.394466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:00.401474 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.401448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"
Sep 16 12:03:00.410647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.410560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsqll" event={"ID":"832aad20-0733-43cf-9a64-e417f6d13bdb","Type":"ContainerStarted","Data":"2d25f609f0e51a8f09a0d7ed7ceb499f6ebe7a174ab7d86576c278826826befe"}
Sep 16 12:03:00.413594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.413421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cd696578-lsfwf" event={"ID":"4a892003-41c7-4f2a-9b29-3c9fad4b0f83","Type":"ContainerStarted","Data":"27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094"}
Sep 16 12:03:00.413594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.413472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cd696578-lsfwf" event={"ID":"4a892003-41c7-4f2a-9b29-3c9fad4b0f83","Type":"ContainerStarted","Data":"3dbdeea520d5e25f83ddbba04d91bb8852a956ff19b4baa9a141ccec9bfce2d9"}
Sep 16 12:03:00.413594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.413547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:03:00.415402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.415228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"
Sep 16 12:03:00.415621 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.415600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-92vxd" event={"ID":"f59f7d0f-6700-43b7-b5f5-726f22dc03ea","Type":"ContainerStarted","Data":"6a7a252db55750a677841bde4e59127b9c55b0c57a06882423807b98105e1478"}
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-ca-trust-extracted\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-bound-sa-token\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-image-registry-private-configuration\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-registry-certificates\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-installation-pull-secrets\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-registry-tls\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-trusted-ca\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bnk\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-kube-api-access-g9bnk\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.418597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-ca-trust-extracted\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.420614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.419360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-registry-certificates\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.421122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.420880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-trusted-ca\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.424605 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.424500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnxvx" event={"ID":"78b0b391-4e63-4177-a34c-c226ca5bc797","Type":"ContainerStarted","Data":"5ae0601f63e39822ba60d77a17ee583bfc930777ac95ef044d80fbc6a02ce713"}
Sep 16 12:03:00.424734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.424711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-installation-pull-secrets\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.425509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.425474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-image-registry-private-configuration\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.427984 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.427942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-registry-tls\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.438389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.438319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bnk\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-kube-api-access-g9bnk\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.438929 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.438890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8cc093d-c95e-463e-bba0-c9f1b57a2ac5-bound-sa-token\") pod \"image-registry-6958b676fb-fvx7b\" (UID: \"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5\") " pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.455010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.453933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67cd696578-lsfwf" podStartSLOduration=57.45391043 podStartE2EDuration="57.45391043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:00.45284831 +0000 UTC m=+67.864865208" watchObservedRunningTime="2026-09-16 12:03:00.45391043 +0000 UTC m=+67.865927333"
Sep 16 12:03:00.497915 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.494176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:00.536000 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.535701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:00.588417 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.588358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:00.693838 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.693805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:00.708047 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.707938    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx"]
Sep 16 12:03:00.712346 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:03:00.712315    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48e1307d_e38f_4713_bf53_891fb47b8094.slice/crio-b001aa9c98aba02af4dd994aef90f728bb23a060bf1662d8419e9c4b8dab9431 WatchSource:0}: Error finding container b001aa9c98aba02af4dd994aef90f728bb23a060bf1662d8419e9c4b8dab9431: Status 404 returned error can't find the container with id b001aa9c98aba02af4dd994aef90f728bb23a060bf1662d8419e9c4b8dab9431
Sep 16 12:03:00.734699 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.734644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4"]
Sep 16 12:03:00.737356 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:03:00.737323    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb102341d_cead_4a14_880e_3c6f5657fbfb.slice/crio-4973d6d57b945e4811ff898bf15ab3c25da94bfc657cb72998c2a1385bfb82ac WatchSource:0}: Error finding container 4973d6d57b945e4811ff898bf15ab3c25da94bfc657cb72998c2a1385bfb82ac: Status 404 returned error can't find the container with id 4973d6d57b945e4811ff898bf15ab3c25da94bfc657cb72998c2a1385bfb82ac
Sep 16 12:03:00.827586 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:00.827525    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6958b676fb-fvx7b"]
Sep 16 12:03:00.834636 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:03:00.834431    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cc093d_c95e_463e_bba0_c9f1b57a2ac5.slice/crio-b9e4f26b666dade361e985e078069ccd11cc7a24fa64b4087c79af8c4c0f489f WatchSource:0}: Error finding container b9e4f26b666dade361e985e078069ccd11cc7a24fa64b4087c79af8c4c0f489f: Status 404 returned error can't find the container with id b9e4f26b666dade361e985e078069ccd11cc7a24fa64b4087c79af8c4c0f489f
Sep 16 12:03:01.435873 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.435790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4" event={"ID":"b102341d-cead-4a14-880e-3c6f5657fbfb","Type":"ContainerStarted","Data":"4973d6d57b945e4811ff898bf15ab3c25da94bfc657cb72998c2a1385bfb82ac"}
Sep 16 12:03:01.439925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.439551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnxvx" event={"ID":"78b0b391-4e63-4177-a34c-c226ca5bc797","Type":"ContainerStarted","Data":"00616b757a1691f56824ddad60fb71f7adbda0551c61e1bea50cabbdededd9ce"}
Sep 16 12:03:01.442602 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.442554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b" event={"ID":"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5","Type":"ContainerStarted","Data":"20740bf517d29b282486c08bffc57188b87ff58a2848e3d1ea75f500d15c4dc2"}
Sep 16 12:03:01.442812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.442618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b" event={"ID":"d8cc093d-c95e-463e-bba0-c9f1b57a2ac5","Type":"ContainerStarted","Data":"b9e4f26b666dade361e985e078069ccd11cc7a24fa64b4087c79af8c4c0f489f"}
Sep 16 12:03:01.443210 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.443091    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:01.445888 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.445864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx" event={"ID":"48e1307d-e38f-4713-bf53-891fb47b8094","Type":"ContainerStarted","Data":"b001aa9c98aba02af4dd994aef90f728bb23a060bf1662d8419e9c4b8dab9431"}
Sep 16 12:03:01.461172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.460061    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lnxvx" podStartSLOduration=67.504092848 podStartE2EDuration="1m8.460040898s" totalImagesPullingTime="955.94805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:01.45856265 +0000 UTC m=+68.870579560" watchObservedRunningTime="2026-09-16 12:03:01.460040898 +0000 UTC m=+68.872057795"
Sep 16 12:03:01.505517 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.505186    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b" podStartSLOduration=1.5051620959999998 podStartE2EDuration="1.505162096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:01.503870662 +0000 UTC m=+68.915887559" watchObservedRunningTime="2026-09-16 12:03:01.505162096 +0000 UTC m=+68.917178997"
Sep 16 12:03:01.650593 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.650474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:01.727327 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.727244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:01.785369 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.785286    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:01.950423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:01.950354    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:02.198309 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:02.198238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:02.301228 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:02.301152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:02.409535 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:02.409501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:02.525828 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:02.525733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:02.586467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:02.586406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:03.538516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:03.538478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:03.744150 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:03.744119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:03.950190 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:03.950150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:03:03.952742 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:03.952717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a467175a-cb94-403f-a278-ce1af1396e85-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-c4qlf\" (UID: \"a467175a-cb94-403f-a278-ce1af1396e85\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:03:04.157622 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.157582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:04.176364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.176326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-tqkcc\""
Sep 16 12:03:04.183655 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.183631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"
Sep 16 12:03:04.258049 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.257974    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:04.359865 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.359830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:04.683386 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.683351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:04.786595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.786545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:04.853766 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:04.853735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:05.013092 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:05.013005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:05.836881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:05.836844    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:07.705661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:07.705628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf"]
Sep 16 12:03:08.482440 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.482404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsqll" event={"ID":"832aad20-0733-43cf-9a64-e417f6d13bdb","Type":"ContainerStarted","Data":"8f6e501a345042e026a45ba760d139b0861bbdff9e1b7e20ce78377fb8d3f3f0"}
Sep 16 12:03:08.482641 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.482465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vsqll" event={"ID":"832aad20-0733-43cf-9a64-e417f6d13bdb","Type":"ContainerStarted","Data":"d703395c436aaaa4f601ce52c422270e13dae39bd7fcde7f671863960b49efec"}
Sep 16 12:03:08.482641 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.482539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vsqll"
Sep 16 12:03:08.484119 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.484087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-92vxd" event={"ID":"f59f7d0f-6700-43b7-b5f5-726f22dc03ea","Type":"ContainerStarted","Data":"3eb89614556ca086369a18beb2e27d6434b792bb5d8232b96ee4304b8769fd9e"}
Sep 16 12:03:08.484244 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.484146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:03:08.485447 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.485403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf" event={"ID":"a467175a-cb94-403f-a278-ce1af1396e85","Type":"ContainerStarted","Data":"26374486009cc167f7f3ad24346c6455b495e3805052b67689e8b416935e9948"}
Sep 16 12:03:08.487062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.487043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lc92b" event={"ID":"bdf83e47-967c-46a9-8e56-ac2a7b147088","Type":"ContainerStarted","Data":"57dc57f33a56520cccd08e9988ea12784648aa1158b4247166ca4ba1184428f4"}
Sep 16 12:03:08.488707 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.488666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4" event={"ID":"b102341d-cead-4a14-880e-3c6f5657fbfb","Type":"ContainerStarted","Data":"f48254f2c966eb4a6cfb7a783f00bbd2a7d555618663c6be4720d9bb5b844d9a"}
Sep 16 12:03:08.490773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.490751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx" event={"ID":"48e1307d-e38f-4713-bf53-891fb47b8094","Type":"ContainerStarted","Data":"48527bc64c025387cc7bc610fc750d89c45d0696b4e8fa12e0d43cbb2ae25a70"}
Sep 16 12:03:08.490856 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.490782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx" event={"ID":"48e1307d-e38f-4713-bf53-891fb47b8094","Type":"ContainerStarted","Data":"475f6b63fe792c2b0761d0b4b6a0baa42d6d388b22807171cb5425608add6bec"}
Sep 16 12:03:08.490856 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.490793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx" event={"ID":"48e1307d-e38f-4713-bf53-891fb47b8094","Type":"ContainerStarted","Data":"8b675afd2a700a95a45aa1d39eee7d9cf240007a72ee4e33aa4606df9c5c2007"}
Sep 16 12:03:08.501182 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.501133    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vsqll" podStartSLOduration=34.012837388 podStartE2EDuration="41.501115114s" totalImagesPullingTime="7.488277726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:08.498840376 +0000 UTC m=+75.910857271" watchObservedRunningTime="2026-09-16 12:03:08.501115114 +0000 UTC m=+75.913132011"
Sep 16 12:03:08.530773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.530680    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lc92b" podStartSLOduration=67.410296847 podStartE2EDuration="1m15.530664093s" totalImagesPullingTime="8.120367246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:08.529991792 +0000 UTC m=+75.942008729" watchObservedRunningTime="2026-09-16 12:03:08.530664093 +0000 UTC m=+75.942680992"
Sep 16 12:03:08.531333 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.531292    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b64d9cb46-7cwkx" podStartSLOduration=1.795228381 podStartE2EDuration="8.531279969s" totalImagesPullingTime="6.736051588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:08.516274724 +0000 UTC m=+75.928291616" watchObservedRunningTime="2026-09-16 12:03:08.531279969 +0000 UTC m=+75.943296868"
Sep 16 12:03:08.548985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.548942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-92vxd" podStartSLOduration=67.704784951 podStartE2EDuration="1m15.548924247s" totalImagesPullingTime="7.844139296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:08.547533687 +0000 UTC m=+75.959550584" watchObservedRunningTime="2026-09-16 12:03:08.548924247 +0000 UTC m=+75.960941144"
Sep 16 12:03:08.563091 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.563049    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd598b9b8-qwvt4" podStartSLOduration=1.8618606 podStartE2EDuration="8.563034982s" totalImagesPullingTime="6.701174382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:08.562372647 +0000 UTC m=+75.974389545" watchObservedRunningTime="2026-09-16 12:03:08.563034982 +0000 UTC m=+75.975051868"
Sep 16 12:03:08.947874 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:08.947844    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:09.005610 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:09.005555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:09.495254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:09.495220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf" event={"ID":"a467175a-cb94-403f-a278-ce1af1396e85","Type":"ContainerStarted","Data":"a558239eaeb43532f7273c5f43f29240c495c3dd92fb7784743b35427bd7edb7"}
Sep 16 12:03:09.511245 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:09.511200    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-c4qlf" podStartSLOduration=36.3663212 podStartE2EDuration="37.511186756s" totalImagesPullingTime="1.144865556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:09.510768999 +0000 UTC m=+76.922785898" watchObservedRunningTime="2026-09-16 12:03:09.511186756 +0000 UTC m=+76.923203914"
Sep 16 12:03:09.641482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:09.641452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:09.738741 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:09.738697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:10.568629 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:10.568598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:10.669259 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:10.669233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:10.887469 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:10.887438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:10.960879 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:10.960832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:13.253371 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:13.253340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:14.684337 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:14.684303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:14.784596 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:14.784547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:14.840204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:14.840173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:15.039405 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.039328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:15.276142 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.276015    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:15.327910 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.327823    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-64qhf"]
Sep 16 12:03:15.330988 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.330967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.333797 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.333776    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:03:15.333797 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.333790    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:03:15.333943 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.333836    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:03:15.335028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.335011    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:03:15.335103 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.335013    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:03:15.335282 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.335267    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twxkx\""
Sep 16 12:03:15.335346 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.335297    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:03:15.374734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.374704    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:15.444897 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.444854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-wtmp\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.444897 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.444893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-root\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.445093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.444911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.445093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.444957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-textfile\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.445093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.445003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-sys\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.445093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.445036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2rk\" (UniqueName: \"kubernetes.io/projected/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-kube-api-access-hl2rk\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.445093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.445058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.445093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.445082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-tls\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.445273 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.445105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-metrics-client-ca\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.453683 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.453662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:15.546337 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2rk\" (UniqueName: \"kubernetes.io/projected/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-kube-api-access-hl2rk\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-tls\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-metrics-client-ca\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-wtmp\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-root\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-textfile\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-sys\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-wtmp\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546940 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-sys\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.546940 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-root\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.547018 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.546990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-textfile\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.547213 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.547187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-metrics-client-ca\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.547276 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.547252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.548802 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.548784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-tls\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.548889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.548842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.555147 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.555106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:15.558004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.557985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2rk\" (UniqueName: \"kubernetes.io/projected/b37dc13c-89f9-4f1d-8f1b-80924302c0e1-kube-api-access-hl2rk\") pod \"node-exporter-64qhf\" (UID: \"b37dc13c-89f9-4f1d-8f1b-80924302c0e1\") " pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.641326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.641290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-64qhf"
Sep 16 12:03:15.652099 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:03:15.652076    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37dc13c_89f9_4f1d_8f1b_80924302c0e1.slice/crio-e76ae13d5cdc765cb856ff3dbc803e5b6de4c78a011977aef7ec36af14463fa9 WatchSource:0}: Error finding container e76ae13d5cdc765cb856ff3dbc803e5b6de4c78a011977aef7ec36af14463fa9: Status 404 returned error can't find the container with id e76ae13d5cdc765cb856ff3dbc803e5b6de4c78a011977aef7ec36af14463fa9
Sep 16 12:03:15.653443 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.653419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:15.755937 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:15.755900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:16.307256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:16.307224    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:16.521942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:16.521911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-64qhf" event={"ID":"b37dc13c-89f9-4f1d-8f1b-80924302c0e1","Type":"ContainerStarted","Data":"e76ae13d5cdc765cb856ff3dbc803e5b6de4c78a011977aef7ec36af14463fa9"}
Sep 16 12:03:16.630711 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:16.630675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:16.724676 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:16.724627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:16.967134 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:16.967081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:17.068440 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:17.068411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:17.276062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:17.275968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:17.378404 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:17.378364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:17.477551 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:17.477510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:17.526801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:17.526719    2577 generic.go:356] "Generic (PLEG): container finished" podID="b37dc13c-89f9-4f1d-8f1b-80924302c0e1" containerID="7c3f44da84d1fa6b006a8423ca0753e52624112080469718c7d116b3d92ddd31" exitCode=0
Sep 16 12:03:17.526801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:17.526778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-64qhf" event={"ID":"b37dc13c-89f9-4f1d-8f1b-80924302c0e1","Type":"ContainerDied","Data":"7c3f44da84d1fa6b006a8423ca0753e52624112080469718c7d116b3d92ddd31"}
Sep 16 12:03:18.237456 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.237421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:18.308412 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.308361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:18.497789 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.497709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vsqll"
Sep 16 12:03:18.532630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.532592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-64qhf" event={"ID":"b37dc13c-89f9-4f1d-8f1b-80924302c0e1","Type":"ContainerStarted","Data":"a23f9f05b89ce47eee02c18c0720df1c1846ddac09ee2687503c269c63cf763e"}
Sep 16 12:03:18.532812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.532653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-64qhf" event={"ID":"b37dc13c-89f9-4f1d-8f1b-80924302c0e1","Type":"ContainerStarted","Data":"6c09b7d6807d5fa2b04e42f8d1af72d9cb6de5a723226230b2c4fcf0250004cb"}
Sep 16 12:03:18.575666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.575562    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-64qhf" podStartSLOduration=2.326398695 podStartE2EDuration="3.575545921s" totalImagesPullingTime="1.186983962s" totalInitContainerRuntime="62.163264ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:18.574688077 +0000 UTC m=+85.986704972" watchObservedRunningTime="2026-09-16 12:03:18.575545921 +0000 UTC m=+85.987562816"
Sep 16 12:03:18.582554 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.582504    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:18.666398 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:18.666364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:20.057648 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:20.057617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:20.165461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:20.165427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:20.451468 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:20.451432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:20.540886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:20.540856    2577 patch_prober.go:28] interesting pod/image-registry-6958b676fb-fvx7b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:03:20.541043 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:20.540903    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b" podUID="d8cc093d-c95e-463e-bba0-c9f1b57a2ac5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:03:20.726176 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:20.726090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:20.786739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:20.786712    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:21.451469 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:21.451441    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:03:21.540468 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:21.540428    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:23.463930 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:23.463903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6958b676fb-fvx7b"
Sep 16 12:03:23.731748 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:23.731667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:23.795619 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:23.795592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:24.473661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:24.473632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:24.533474 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:24.533450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:24.923324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:24.923296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:24.978347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:24.978318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:25.422036 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:25.421998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:25.521055 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:25.521023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:26.475288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.475248    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67cd696578-lsfwf" podUID="4a892003-41c7-4f2a-9b29-3c9fad4b0f83" containerName="registry" containerID="cri-o://27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094" gracePeriod=30
Sep 16 12:03:26.711290 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.711265    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:03:26.836310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-installation-pull-secrets\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.836310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.836310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-image-registry-private-configuration\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.836310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6b66\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-kube-api-access-w6b66\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.836668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-bound-sa-token\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.836668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-ca-trust-extracted\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.836668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-certificates\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.836668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-trusted-ca\") pod \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\" (UID: \"4a892003-41c7-4f2a-9b29-3c9fad4b0f83\") "
Sep 16 12:03:26.837015 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.836959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:03:26.837153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.837057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:03:26.838934 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.838866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:03:26.839118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.839084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-kube-api-access-w6b66" (OuterVolumeSpecName: "kube-api-access-w6b66") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "kube-api-access-w6b66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:03:26.839226 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.839123    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:03:26.839226 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.839185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:03:26.839226 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.839195    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:03:26.844401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.844375    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4a892003-41c7-4f2a-9b29-3c9fad4b0f83" (UID: "4a892003-41c7-4f2a-9b29-3c9fad4b0f83"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:03:26.937557 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937520    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-installation-pull-secrets\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:26.937557 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937550    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:26.937557 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937561    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-image-registry-private-configuration\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:26.937801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937593    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6b66\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-kube-api-access-w6b66\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:26.937801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937602    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-bound-sa-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:26.937801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937612    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-ca-trust-extracted\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:26.937801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937619    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-registry-certificates\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:26.937801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:26.937627    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a892003-41c7-4f2a-9b29-3c9fad4b0f83-trusted-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:03:27.564322 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.564286    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a892003-41c7-4f2a-9b29-3c9fad4b0f83" containerID="27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094" exitCode=0
Sep 16 12:03:27.564495 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.564351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cd696578-lsfwf"
Sep 16 12:03:27.564495 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.564350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cd696578-lsfwf" event={"ID":"4a892003-41c7-4f2a-9b29-3c9fad4b0f83","Type":"ContainerDied","Data":"27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094"}
Sep 16 12:03:27.564495 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.564462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cd696578-lsfwf" event={"ID":"4a892003-41c7-4f2a-9b29-3c9fad4b0f83","Type":"ContainerDied","Data":"3dbdeea520d5e25f83ddbba04d91bb8852a956ff19b4baa9a141ccec9bfce2d9"}
Sep 16 12:03:27.564495 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.564481    2577 scope.go:117] "RemoveContainer" containerID="27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094"
Sep 16 12:03:27.572988 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.572968    2577 scope.go:117] "RemoveContainer" containerID="27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094"
Sep 16 12:03:27.573310 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:03:27.573283    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094\": container with ID starting with 27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094 not found: ID does not exist" containerID="27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094"
Sep 16 12:03:27.573411 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.573316    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094"} err="failed to get container status \"27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094\": rpc error: code = NotFound desc = could not find container \"27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094\": container with ID starting with 27979ee4d1c8dc7055de3c750dd6967af83b432dc3a994200a89d1c0775c2094 not found: ID does not exist"
Sep 16 12:03:27.582045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.582019    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67cd696578-lsfwf"]
Sep 16 12:03:27.586234 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.586185    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67cd696578-lsfwf"]
Sep 16 12:03:27.764337 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.764305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:27.821828 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:27.821738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:29.109457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:29.109421    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a892003-41c7-4f2a-9b29-3c9fad4b0f83" path="/var/lib/kubelet/pods/4a892003-41c7-4f2a-9b29-3c9fad4b0f83/volumes"
Sep 16 12:03:29.372315 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:29.372224    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:29.472341 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:29.472311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:30.137349 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:30.137301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:30.190143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:30.190114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:32.372544 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:32.372503    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:32.433935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:32.433899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:33.513207 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:33.513152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:33.614173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:33.614137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:36.852273 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:36.852235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:36.930261 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:36.930228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:39.498089 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:39.498059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-92vxd"
Sep 16 12:03:40.447450 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:40.447415    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:40.516377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:40.516340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:42.667510 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:42.667473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:42.719614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:42.719563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:42.774655 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:42.774620    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:55.432954 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:55.432919    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:55.534132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:55.534101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:56.745168 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:56.745138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:03:59.049247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:03:59.049208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:31.399675 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:31.399637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:31.516853 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:31.516818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:37.353426 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:37.353395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:37.476385 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:37.476345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:37.544920 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:37.544885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:38.066951 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:38.066916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:38.119153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:38.119122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:43.299727 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:43.299689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:43.520531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:43.520488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:47.925669 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:47.925554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:48.026094 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:48.026060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:48.081968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:48.081940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:49.777706 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:49.777670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:55.428808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:55.428775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:04:55.529915 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:04:55.529881    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:05:41.233506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:05:41.233473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:05:46.140584 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:05:46.140531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:05:46.234344 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:05:46.234305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:12.915950 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:12.915912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:13.754337 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:13.754288    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:14.920027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.919948    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"]
Sep 16 12:06:14.920450 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.920305    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a892003-41c7-4f2a-9b29-3c9fad4b0f83" containerName="registry"
Sep 16 12:06:14.920450 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.920317    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a892003-41c7-4f2a-9b29-3c9fad4b0f83" containerName="registry"
Sep 16 12:06:14.920450 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.920364    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a892003-41c7-4f2a-9b29-3c9fad4b0f83" containerName="registry"
Sep 16 12:06:14.922110 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.922096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:14.924831 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.924805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 12:06:14.924968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.924835    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:06:14.924968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.924930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p8726\""
Sep 16 12:06:14.935604 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.935579    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"]
Sep 16 12:06:14.997983 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.997946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqbsh\" (UniqueName: \"kubernetes.io/projected/dcd6b4f8-d677-4c95-adf1-22233922492a-kube-api-access-nqbsh\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pc5j\" (UID: \"dcd6b4f8-d677-4c95-adf1-22233922492a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:14.998148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:14.998006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcd6b4f8-d677-4c95-adf1-22233922492a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pc5j\" (UID: \"dcd6b4f8-d677-4c95-adf1-22233922492a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:15.098758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:15.098711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqbsh\" (UniqueName: \"kubernetes.io/projected/dcd6b4f8-d677-4c95-adf1-22233922492a-kube-api-access-nqbsh\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pc5j\" (UID: \"dcd6b4f8-d677-4c95-adf1-22233922492a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:15.098854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:15.098789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcd6b4f8-d677-4c95-adf1-22233922492a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pc5j\" (UID: \"dcd6b4f8-d677-4c95-adf1-22233922492a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:15.099113 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:15.099098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcd6b4f8-d677-4c95-adf1-22233922492a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pc5j\" (UID: \"dcd6b4f8-d677-4c95-adf1-22233922492a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:15.107298 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:15.107274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqbsh\" (UniqueName: \"kubernetes.io/projected/dcd6b4f8-d677-4c95-adf1-22233922492a-kube-api-access-nqbsh\") pod \"cert-manager-operator-controller-manager-55c444bf86-6pc5j\" (UID: \"dcd6b4f8-d677-4c95-adf1-22233922492a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:15.231166 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:15.231076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"
Sep 16 12:06:15.498983 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:15.498902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j"]
Sep 16 12:06:15.502529 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:06:15.502489    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd6b4f8_d677_4c95_adf1_22233922492a.slice/crio-3c27f00fda3200afa58f5baf29286c63eb76bc47d3bec067e3724acc9f16abbd WatchSource:0}: Error finding container 3c27f00fda3200afa58f5baf29286c63eb76bc47d3bec067e3724acc9f16abbd: Status 404 returned error can't find the container with id 3c27f00fda3200afa58f5baf29286c63eb76bc47d3bec067e3724acc9f16abbd
Sep 16 12:06:16.111036 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:16.110999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j" event={"ID":"dcd6b4f8-d677-4c95-adf1-22233922492a","Type":"ContainerStarted","Data":"3c27f00fda3200afa58f5baf29286c63eb76bc47d3bec067e3724acc9f16abbd"}
Sep 16 12:06:16.729093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:16.729062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:16.832158 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:16.832122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:21.320332 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:21.320282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:21.373784 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:21.373754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:28.138191 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:28.138147    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:28.152060 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:28.152024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j" event={"ID":"dcd6b4f8-d677-4c95-adf1-22233922492a","Type":"ContainerStarted","Data":"b13274b42a28af7ba988bbb1916de0630fe1741c4bcd479ff4c47b306d0b1378"}
Sep 16 12:06:28.177148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:28.177095    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6pc5j" podStartSLOduration=2.300584304 podStartE2EDuration="14.177081488s" totalImagesPullingTime="11.876497184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:06:28.176656683 +0000 UTC m=+275.588673572" watchObservedRunningTime="2026-09-16 12:06:28.177081488 +0000 UTC m=+275.589098384"
Sep 16 12:06:28.242255 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:28.242223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:28.661034 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:28.661001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:28.767276 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:28.767240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:30.152359 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:30.152337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:30.556505 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:30.556422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:35.471394 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:35.471357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:35.576127 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:35.576096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:38.156150 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:38.156119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:38.958363 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:38.958329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:39.058609 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:39.058562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:40.775870 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.775835    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"]
Sep 16 12:06:40.778498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.778481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:40.781137 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.781109    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:06:40.782266 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.782239    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7t6rb\""
Sep 16 12:06:40.782375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.782247    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:06:40.790376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.789950    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"]
Sep 16 12:06:40.885785 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.885749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66bc3b23-d325-4ff6-b9fe-46fba1560000-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cbksd\" (UID: \"66bc3b23-d325-4ff6-b9fe-46fba1560000\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:40.885956 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.885788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjm7\" (UniqueName: \"kubernetes.io/projected/66bc3b23-d325-4ff6-b9fe-46fba1560000-kube-api-access-2tjm7\") pod \"openshift-lws-operator-fd8ccff4c-cbksd\" (UID: \"66bc3b23-d325-4ff6-b9fe-46fba1560000\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:40.986955 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.986911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66bc3b23-d325-4ff6-b9fe-46fba1560000-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cbksd\" (UID: \"66bc3b23-d325-4ff6-b9fe-46fba1560000\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:40.986955 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.986958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjm7\" (UniqueName: \"kubernetes.io/projected/66bc3b23-d325-4ff6-b9fe-46fba1560000-kube-api-access-2tjm7\") pod \"openshift-lws-operator-fd8ccff4c-cbksd\" (UID: \"66bc3b23-d325-4ff6-b9fe-46fba1560000\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:40.987401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.987374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66bc3b23-d325-4ff6-b9fe-46fba1560000-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cbksd\" (UID: \"66bc3b23-d325-4ff6-b9fe-46fba1560000\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:40.995169 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:40.995138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjm7\" (UniqueName: \"kubernetes.io/projected/66bc3b23-d325-4ff6-b9fe-46fba1560000-kube-api-access-2tjm7\") pod \"openshift-lws-operator-fd8ccff4c-cbksd\" (UID: \"66bc3b23-d325-4ff6-b9fe-46fba1560000\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:41.088830 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:41.088741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"
Sep 16 12:06:41.356067 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:41.355985    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd"]
Sep 16 12:06:41.358798 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:06:41.358772    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66bc3b23_d325_4ff6_b9fe_46fba1560000.slice/crio-33d4d79bb01158a15b46fd45a4e995ebf369b0f4a1014b20781ea48ea30d75c1 WatchSource:0}: Error finding container 33d4d79bb01158a15b46fd45a4e995ebf369b0f4a1014b20781ea48ea30d75c1: Status 404 returned error can't find the container with id 33d4d79bb01158a15b46fd45a4e995ebf369b0f4a1014b20781ea48ea30d75c1
Sep 16 12:06:42.198886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:42.198844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd" event={"ID":"66bc3b23-d325-4ff6-b9fe-46fba1560000","Type":"ContainerStarted","Data":"33d4d79bb01158a15b46fd45a4e995ebf369b0f4a1014b20781ea48ea30d75c1"}
Sep 16 12:06:44.208278 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:44.208240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd" event={"ID":"66bc3b23-d325-4ff6-b9fe-46fba1560000","Type":"ContainerStarted","Data":"d1b3d6828a24d73c96b6d7368630650491f412208a20eb38c87d107fcf9262bb"}
Sep 16 12:06:44.228428 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:44.228375    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cbksd" podStartSLOduration=1.565873608 podStartE2EDuration="4.228361198s" totalImagesPullingTime="2.66248759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:06:44.227228595 +0000 UTC m=+291.639245494" watchObservedRunningTime="2026-09-16 12:06:44.228361198 +0000 UTC m=+291.640378093"
Sep 16 12:06:44.973344 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:44.973308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:45.074091 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:45.074054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:47.090257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:47.090221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:06:53.027159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:06:53.027130    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:07:02.913916 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:02.913886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:04.019962 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.019930    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"]
Sep 16 12:07:04.022176 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.022155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.025108 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.025085    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:07:04.026071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.026048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:07:04.026167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.026097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:07:04.026329 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.026316    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-22d2l\""
Sep 16 12:07:04.026627 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.026611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:07:04.037806 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.037783    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"]
Sep 16 12:07:04.159704 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.159668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6xn\" (UniqueName: \"kubernetes.io/projected/7189d010-0c01-4b6d-be71-a0c45572fcf2-kube-api-access-dv6xn\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.159885 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.159716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7189d010-0c01-4b6d-be71-a0c45572fcf2-manifests\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.159885 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.159772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7189d010-0c01-4b6d-be71-a0c45572fcf2-tmp\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.159885 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.159843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7189d010-0c01-4b6d-be71-a0c45572fcf2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.160027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.159886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7189d010-0c01-4b6d-be71-a0c45572fcf2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.260792 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.260761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7189d010-0c01-4b6d-be71-a0c45572fcf2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.260957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.260801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6xn\" (UniqueName: \"kubernetes.io/projected/7189d010-0c01-4b6d-be71-a0c45572fcf2-kube-api-access-dv6xn\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.260957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.260828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7189d010-0c01-4b6d-be71-a0c45572fcf2-manifests\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.260957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.260843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7189d010-0c01-4b6d-be71-a0c45572fcf2-tmp\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.260957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.260874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7189d010-0c01-4b6d-be71-a0c45572fcf2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.261252 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.261223    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7189d010-0c01-4b6d-be71-a0c45572fcf2-manifests\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.263195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.263162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7189d010-0c01-4b6d-be71-a0c45572fcf2-tmp\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.263290 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.263227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7189d010-0c01-4b6d-be71-a0c45572fcf2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.263290 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.263270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7189d010-0c01-4b6d-be71-a0c45572fcf2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.273117 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.273072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6xn\" (UniqueName: \"kubernetes.io/projected/7189d010-0c01-4b6d-be71-a0c45572fcf2-kube-api-access-dv6xn\") pod \"opendatahub-operator-controller-manager-6b54c497cf-mlx69\" (UID: \"7189d010-0c01-4b6d-be71-a0c45572fcf2\") " pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.287797 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.287763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:04.332368 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.332335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:04.389410 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.389336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:04.493675 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.493643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:04.591208 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.591139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:04.604238 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.604218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"]
Sep 16 12:07:04.606195 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:07:04.606174    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7189d010_0c01_4b6d_be71_a0c45572fcf2.slice/crio-acf4c9046d20d6af8ba48a68f3e97a73adfa46b72ef03c39c5f049c7fa4b8b73 WatchSource:0}: Error finding container acf4c9046d20d6af8ba48a68f3e97a73adfa46b72ef03c39c5f049c7fa4b8b73: Status 404 returned error can't find the container with id acf4c9046d20d6af8ba48a68f3e97a73adfa46b72ef03c39c5f049c7fa4b8b73
Sep 16 12:07:04.607864 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:04.607848    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:07:05.293250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:05.293195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69" event={"ID":"7189d010-0c01-4b6d-be71-a0c45572fcf2","Type":"ContainerStarted","Data":"acf4c9046d20d6af8ba48a68f3e97a73adfa46b72ef03c39c5f049c7fa4b8b73"}
Sep 16 12:07:08.307201 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:08.307164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69" event={"ID":"7189d010-0c01-4b6d-be71-a0c45572fcf2","Type":"ContainerStarted","Data":"32ab73d4025d795e00e46bdf8eea292be7911a54b9534a4c64b35ef4f888d5e7"}
Sep 16 12:07:09.312740 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:09.312699    2577 generic.go:356] "Generic (PLEG): container finished" podID="7189d010-0c01-4b6d-be71-a0c45572fcf2" containerID="32ab73d4025d795e00e46bdf8eea292be7911a54b9534a4c64b35ef4f888d5e7" exitCode=0
Sep 16 12:07:09.313099 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:09.312787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69" event={"ID":"7189d010-0c01-4b6d-be71-a0c45572fcf2","Type":"ContainerDied","Data":"32ab73d4025d795e00e46bdf8eea292be7911a54b9534a4c64b35ef4f888d5e7"}
Sep 16 12:07:10.318666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:10.318632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69" event={"ID":"7189d010-0c01-4b6d-be71-a0c45572fcf2","Type":"ContainerStarted","Data":"08eb7c1ff6e44722a788602167b065a9b9ae7770512ff2bd7a2cd5d369ff9206"}
Sep 16 12:07:10.319059 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:10.318735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:10.341945 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:10.341896    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69" podStartSLOduration=2.991404442 podStartE2EDuration="7.341881356s" totalImagesPullingTime="3.323300849s" totalInitContainerRuntime="1.027176065s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:10.339446945 +0000 UTC m=+317.751463843" watchObservedRunningTime="2026-09-16 12:07:10.341881356 +0000 UTC m=+317.753898251"
Sep 16 12:07:16.915940 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:16.915897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:17.015187 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:17.015153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:21.326094 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:21.326063    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b54c497cf-mlx69"
Sep 16 12:07:23.518354 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:23.518321    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:23.688111 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:23.688057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:24.082148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.082112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:24.117308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.117269    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"]
Sep 16 12:07:24.119474 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.119451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.125263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.125239    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 12:07:24.125381 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.125259    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 12:07:24.126375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.126353    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:07:24.126853 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.126828    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l7wkc\""
Sep 16 12:07:24.133210 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.133193    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:07:24.158999 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.158975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"]
Sep 16 12:07:24.211318 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.211286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970c6ed-7aa3-422a-88b7-f2491a98e453-tls-certs\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.211465 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.211336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4k6\" (UniqueName: \"kubernetes.io/projected/7970c6ed-7aa3-422a-88b7-f2491a98e453-kube-api-access-rx4k6\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.211465 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.211428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7970c6ed-7aa3-422a-88b7-f2491a98e453-tmp\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.312866 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.312835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970c6ed-7aa3-422a-88b7-f2491a98e453-tls-certs\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.312866 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.312873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4k6\" (UniqueName: \"kubernetes.io/projected/7970c6ed-7aa3-422a-88b7-f2491a98e453-kube-api-access-rx4k6\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.313119 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.313073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7970c6ed-7aa3-422a-88b7-f2491a98e453-tmp\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.315171 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.315136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7970c6ed-7aa3-422a-88b7-f2491a98e453-tmp\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.315550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.315527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7970c6ed-7aa3-422a-88b7-f2491a98e453-tls-certs\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.336221 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.336153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4k6\" (UniqueName: \"kubernetes.io/projected/7970c6ed-7aa3-422a-88b7-f2491a98e453-kube-api-access-rx4k6\") pod \"kube-auth-proxy-d74dbbc-hvfvl\" (UID: \"7970c6ed-7aa3-422a-88b7-f2491a98e453\") " pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.381173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.381120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:24.431395 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.431366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"
Sep 16 12:07:24.712174 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:24.712132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl"]
Sep 16 12:07:24.715188 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:07:24.715163    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7970c6ed_7aa3_422a_88b7_f2491a98e453.slice/crio-35efc89dbc51f1f4847e76e468f341d04d2f4b0f89dd94980be37d03bfba5551 WatchSource:0}: Error finding container 35efc89dbc51f1f4847e76e468f341d04d2f4b0f89dd94980be37d03bfba5551: Status 404 returned error can't find the container with id 35efc89dbc51f1f4847e76e468f341d04d2f4b0f89dd94980be37d03bfba5551
Sep 16 12:07:25.379922 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:25.379869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl" event={"ID":"7970c6ed-7aa3-422a-88b7-f2491a98e453","Type":"ContainerStarted","Data":"35efc89dbc51f1f4847e76e468f341d04d2f4b0f89dd94980be37d03bfba5551"}
Sep 16 12:07:25.923917 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:25.923870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:26.023066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:26.023021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:27.241731 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:27.241692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:28.396653 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:28.396621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl" event={"ID":"7970c6ed-7aa3-422a-88b7-f2491a98e453","Type":"ContainerStarted","Data":"903a731c249640acd74ad98543f66be3217e8ccbc204c7eaa61ddf0a1289df50"}
Sep 16 12:07:30.629001 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:30.628948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:30.739190 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:30.739155    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:30.948126 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:30.948039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:31.063204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:31.063165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:31.154702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:31.154657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:33.833781 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:33.833733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:34.222016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.221971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:34.624063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.623946    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d74dbbc-hvfvl" podStartSLOduration=7.957330092 podStartE2EDuration="10.623923008s" totalImagesPullingTime="2.666592916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:28.424826982 +0000 UTC m=+335.836843879" watchObservedRunningTime="2026-09-16 12:07:34.623923008 +0000 UTC m=+342.035939906"
Sep 16 12:07:34.625179 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.625154    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8ps7n"]
Sep 16 12:07:34.627820 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.627798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:34.631084 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.631060    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 12:07:34.631200 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.631079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qcxxs\""
Sep 16 12:07:34.641379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.641352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8ps7n"]
Sep 16 12:07:34.670760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.670729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:34.699433 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.699408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4n5\" (UniqueName: \"kubernetes.io/projected/44725276-0e2d-4c99-b373-748fcf0ab7bc-kube-api-access-jl4n5\") pod \"odh-model-controller-7998c67dc4-8ps7n\" (UID: \"44725276-0e2d-4c99-b373-748fcf0ab7bc\") " pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:34.699560 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.699448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44725276-0e2d-4c99-b373-748fcf0ab7bc-cert\") pod \"odh-model-controller-7998c67dc4-8ps7n\" (UID: \"44725276-0e2d-4c99-b373-748fcf0ab7bc\") " pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:34.732258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.732232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:34.800158 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.800131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4n5\" (UniqueName: \"kubernetes.io/projected/44725276-0e2d-4c99-b373-748fcf0ab7bc-kube-api-access-jl4n5\") pod \"odh-model-controller-7998c67dc4-8ps7n\" (UID: \"44725276-0e2d-4c99-b373-748fcf0ab7bc\") " pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:34.800327 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.800179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44725276-0e2d-4c99-b373-748fcf0ab7bc-cert\") pod \"odh-model-controller-7998c67dc4-8ps7n\" (UID: \"44725276-0e2d-4c99-b373-748fcf0ab7bc\") " pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:34.800392 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:07:34.800327    2577 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 12:07:34.800392 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:07:34.800388    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44725276-0e2d-4c99-b373-748fcf0ab7bc-cert podName:44725276-0e2d-4c99-b373-748fcf0ab7bc nodeName:}" failed. No retries permitted until 2026-09-16 12:07:35.300369818 +0000 UTC m=+342.712386696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/44725276-0e2d-4c99-b373-748fcf0ab7bc-cert") pod "odh-model-controller-7998c67dc4-8ps7n" (UID: "44725276-0e2d-4c99-b373-748fcf0ab7bc") : secret "odh-model-controller-webhook-cert" not found
Sep 16 12:07:34.810983 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.810959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4n5\" (UniqueName: \"kubernetes.io/projected/44725276-0e2d-4c99-b373-748fcf0ab7bc-kube-api-access-jl4n5\") pod \"odh-model-controller-7998c67dc4-8ps7n\" (UID: \"44725276-0e2d-4c99-b373-748fcf0ab7bc\") " pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:34.833764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.833734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:34.986454 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:34.986419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:35.183341 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.183301    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vz5nf"]
Sep 16 12:07:35.185409 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.185394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.188227 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.188197    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 12:07:35.197325 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.197301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vz5nf"]
Sep 16 12:07:35.204071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.204043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvh9\" (UniqueName: \"kubernetes.io/projected/50418aa1-661f-4523-b5d3-031f0448be72-kube-api-access-5lvh9\") pod \"model-serving-api-54d89bd5fc-vz5nf\" (UID: \"50418aa1-661f-4523-b5d3-031f0448be72\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.204198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.204100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50418aa1-661f-4523-b5d3-031f0448be72-tls-certs\") pod \"model-serving-api-54d89bd5fc-vz5nf\" (UID: \"50418aa1-661f-4523-b5d3-031f0448be72\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.214775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.214735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:35.305498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.305414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvh9\" (UniqueName: \"kubernetes.io/projected/50418aa1-661f-4523-b5d3-031f0448be72-kube-api-access-5lvh9\") pod \"model-serving-api-54d89bd5fc-vz5nf\" (UID: \"50418aa1-661f-4523-b5d3-031f0448be72\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.305661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.305558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50418aa1-661f-4523-b5d3-031f0448be72-tls-certs\") pod \"model-serving-api-54d89bd5fc-vz5nf\" (UID: \"50418aa1-661f-4523-b5d3-031f0448be72\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.305661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.305618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44725276-0e2d-4c99-b373-748fcf0ab7bc-cert\") pod \"odh-model-controller-7998c67dc4-8ps7n\" (UID: \"44725276-0e2d-4c99-b373-748fcf0ab7bc\") " pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:35.305738 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:07:35.305698    2577 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 12:07:35.305771 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:07:35.305758    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50418aa1-661f-4523-b5d3-031f0448be72-tls-certs podName:50418aa1-661f-4523-b5d3-031f0448be72 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:35.805744071 +0000 UTC m=+343.217760945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/50418aa1-661f-4523-b5d3-031f0448be72-tls-certs") pod "model-serving-api-54d89bd5fc-vz5nf" (UID: "50418aa1-661f-4523-b5d3-031f0448be72") : secret "model-serving-api-tls" not found
Sep 16 12:07:35.308195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.308174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44725276-0e2d-4c99-b373-748fcf0ab7bc-cert\") pod \"odh-model-controller-7998c67dc4-8ps7n\" (UID: \"44725276-0e2d-4c99-b373-748fcf0ab7bc\") " pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:35.316411 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.316382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvh9\" (UniqueName: \"kubernetes.io/projected/50418aa1-661f-4523-b5d3-031f0448be72-kube-api-access-5lvh9\") pod \"model-serving-api-54d89bd5fc-vz5nf\" (UID: \"50418aa1-661f-4523-b5d3-031f0448be72\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.408739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.408689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:35.538604 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.538526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:35.809258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.809217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50418aa1-661f-4523-b5d3-031f0448be72-tls-certs\") pod \"model-serving-api-54d89bd5fc-vz5nf\" (UID: \"50418aa1-661f-4523-b5d3-031f0448be72\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.812225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.812196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50418aa1-661f-4523-b5d3-031f0448be72-tls-certs\") pod \"model-serving-api-54d89bd5fc-vz5nf\" (UID: \"50418aa1-661f-4523-b5d3-031f0448be72\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:35.814296 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:35.814274    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8ps7n"]
Sep 16 12:07:35.816294 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:07:35.816264    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44725276_0e2d_4c99_b373_748fcf0ab7bc.slice/crio-57c025807b9f583f31b3b6b1eee80769adf178000413c61ccc51334c5f2238a7 WatchSource:0}: Error finding container 57c025807b9f583f31b3b6b1eee80769adf178000413c61ccc51334c5f2238a7: Status 404 returned error can't find the container with id 57c025807b9f583f31b3b6b1eee80769adf178000413c61ccc51334c5f2238a7
Sep 16 12:07:36.096095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:36.096055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:36.375102 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:36.375021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vz5nf"]
Sep 16 12:07:36.379516 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:07:36.379490    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50418aa1_661f_4523_b5d3_031f0448be72.slice/crio-1f4b4cedc85b894f93d7ce1bd7a91d82d1a3580e0464d8ffa580ae1868323cb2 WatchSource:0}: Error finding container 1f4b4cedc85b894f93d7ce1bd7a91d82d1a3580e0464d8ffa580ae1868323cb2: Status 404 returned error can't find the container with id 1f4b4cedc85b894f93d7ce1bd7a91d82d1a3580e0464d8ffa580ae1868323cb2
Sep 16 12:07:36.429072 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:36.429038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf" event={"ID":"50418aa1-661f-4523-b5d3-031f0448be72","Type":"ContainerStarted","Data":"1f4b4cedc85b894f93d7ce1bd7a91d82d1a3580e0464d8ffa580ae1868323cb2"}
Sep 16 12:07:36.430366 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:36.430328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n" event={"ID":"44725276-0e2d-4c99-b373-748fcf0ab7bc","Type":"ContainerStarted","Data":"57c025807b9f583f31b3b6b1eee80769adf178000413c61ccc51334c5f2238a7"}
Sep 16 12:07:39.445436 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.445395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf" event={"ID":"50418aa1-661f-4523-b5d3-031f0448be72","Type":"ContainerStarted","Data":"510954155fc981bc20fc1d7d94312e79716b8b3c5def6a016afc60eac77776e7"}
Sep 16 12:07:39.445951 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.445590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:07:39.447179 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.447122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n" event={"ID":"44725276-0e2d-4c99-b373-748fcf0ab7bc","Type":"ContainerStarted","Data":"cf903d66f3cf869b4c34b2b3bfe8c7d5fdea5a1228bb6bbbf45282ffe410de32"}
Sep 16 12:07:39.447300 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.447221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:39.461524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.461435    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf" podStartSLOduration=1.639220685 podStartE2EDuration="4.461421051s" totalImagesPullingTime="2.822200366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:39.460928878 +0000 UTC m=+346.872945774" watchObservedRunningTime="2026-09-16 12:07:39.461421051 +0000 UTC m=+346.873437947"
Sep 16 12:07:39.477364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.477295    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n" podStartSLOduration=2.065600092 podStartE2EDuration="5.477255829s" totalImagesPullingTime="3.411655737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:39.475975911 +0000 UTC m=+346.887992809" watchObservedRunningTime="2026-09-16 12:07:39.477255829 +0000 UTC m=+346.889272726"
Sep 16 12:07:39.635343 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.635308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:39.690027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:39.689992    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:41.746296 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.746261    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"]
Sep 16 12:07:41.752770 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.752751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.755814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.755790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 12:07:41.756005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.755987    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 12:07:41.756132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.756109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5z849\""
Sep 16 12:07:41.756466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.756450    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 12:07:41.756581 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.756554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:07:41.784035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.784008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"]
Sep 16 12:07:41.862026 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.862000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.862173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.862030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5k2f\" (UniqueName: \"kubernetes.io/projected/f51eb83f-4f25-4a0a-8603-95d2f58e5632-kube-api-access-k5k2f\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.862173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.862054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.862173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.862075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.862173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.862144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51eb83f-4f25-4a0a-8603-95d2f58e5632-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.862305 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.862183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.862305 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.862207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.963487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.963449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.963487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.963490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.963745 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.963531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.963745 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.963546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5k2f\" (UniqueName: \"kubernetes.io/projected/f51eb83f-4f25-4a0a-8603-95d2f58e5632-kube-api-access-k5k2f\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.963745 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.963586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.963745 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.963616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.963745 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.963659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51eb83f-4f25-4a0a-8603-95d2f58e5632-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.964352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.964320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.965946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.965922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51eb83f-4f25-4a0a-8603-95d2f58e5632-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.966144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.966121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.966144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.966139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.966250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.966177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.972274 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.972254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5k2f\" (UniqueName: \"kubernetes.io/projected/f51eb83f-4f25-4a0a-8603-95d2f58e5632-kube-api-access-k5k2f\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:41.974396 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:41.974376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51eb83f-4f25-4a0a-8603-95d2f58e5632-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dlp49\" (UID: \"f51eb83f-4f25-4a0a-8603-95d2f58e5632\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:42.061605 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:42.061518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:42.378198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:42.378080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"]
Sep 16 12:07:42.381927 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:07:42.381892    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51eb83f_4f25_4a0a_8603_95d2f58e5632.slice/crio-2bb928783827ae754eb3ca991d0ec924421f9a20aa00d9f8f7237fe8dc7f7b56 WatchSource:0}: Error finding container 2bb928783827ae754eb3ca991d0ec924421f9a20aa00d9f8f7237fe8dc7f7b56: Status 404 returned error can't find the container with id 2bb928783827ae754eb3ca991d0ec924421f9a20aa00d9f8f7237fe8dc7f7b56
Sep 16 12:07:42.461492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:42.461451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49" event={"ID":"f51eb83f-4f25-4a0a-8603-95d2f58e5632","Type":"ContainerStarted","Data":"2bb928783827ae754eb3ca991d0ec924421f9a20aa00d9f8f7237fe8dc7f7b56"}
Sep 16 12:07:44.382361 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:44.382303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:44.475895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:44.475864    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:44.779593 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:44.779496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:44.991758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:44.991717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:45.664270 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:45.664165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:45.786328 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:45.786293    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:45.878054 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:45.878021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:07:46.930299 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:46.930263    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:07:46.930555 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:46.930334    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:07:47.487437 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:47.487398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49" event={"ID":"f51eb83f-4f25-4a0a-8603-95d2f58e5632","Type":"ContainerStarted","Data":"f5e6f85fbc1f266fde31f89652f25dafbf55bcddcbab2a6497eebf6ef2794f72"}
Sep 16 12:07:47.487620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:47.487600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:47.489288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:47.489260    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-dlp49 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:07:47.489401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:47.489327    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49" podUID="f51eb83f-4f25-4a0a-8603-95d2f58e5632" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:07:47.521910 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:47.521859    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49" podStartSLOduration=1.975814314 podStartE2EDuration="6.521839716s" totalImagesPullingTime="4.546025402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:47.520027475 +0000 UTC m=+354.932044370" watchObservedRunningTime="2026-09-16 12:07:47.521839716 +0000 UTC m=+354.933856613"
Sep 16 12:07:48.492550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:48.492521    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dlp49"
Sep 16 12:07:50.453341 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:50.453312    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8ps7n"
Sep 16 12:07:50.455204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:07:50.455181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vz5nf"
Sep 16 12:08:21.842727 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:21.842696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:21.940025 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:21.939995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:22.040772 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:22.040732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:22.518953 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:22.518907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:22.571851 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:22.571814    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:32.354721 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:32.354676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:32.450108 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:32.450072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:32.507253 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:32.507214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:35.644774 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:35.644735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:35.747242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:35.747207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:41.995341 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:41.995304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:47.095955 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:47.095922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:49.886062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:49.886030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:51.139089 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.139058    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"]
Sep 16 12:08:51.142144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.142127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:08:51.147096 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.147073    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7tzx8\""
Sep 16 12:08:51.148513 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.148496    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:08:51.148598 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.148536    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:08:51.163021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.163001    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"]
Sep 16 12:08:51.217215 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.217185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/b5b9ca52-e5c5-4235-b27b-197053c15c7d-kube-api-access-rwdv9\") pod \"limitador-operator-controller-manager-74495c69f8-vnrft\" (UID: \"b5b9ca52-e5c5-4235-b27b-197053c15c7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:08:51.317739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.317705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/b5b9ca52-e5c5-4235-b27b-197053c15c7d-kube-api-access-rwdv9\") pod \"limitador-operator-controller-manager-74495c69f8-vnrft\" (UID: \"b5b9ca52-e5c5-4235-b27b-197053c15c7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:08:51.329658 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.329626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/b5b9ca52-e5c5-4235-b27b-197053c15c7d-kube-api-access-rwdv9\") pod \"limitador-operator-controller-manager-74495c69f8-vnrft\" (UID: \"b5b9ca52-e5c5-4235-b27b-197053c15c7d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:08:51.451972 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.451896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:08:51.486740 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.486709    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:51.725469 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.725437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"]
Sep 16 12:08:51.728015 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:08:51.727979    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b9ca52_e5c5_4235_b27b_197053c15c7d.slice/crio-1f8b92e36c475789649ef7010828cd4d90f3a0ee40623faf2f892713eacfc5fa WatchSource:0}: Error finding container 1f8b92e36c475789649ef7010828cd4d90f3a0ee40623faf2f892713eacfc5fa: Status 404 returned error can't find the container with id 1f8b92e36c475789649ef7010828cd4d90f3a0ee40623faf2f892713eacfc5fa
Sep 16 12:08:51.759794 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.759759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft" event={"ID":"b5b9ca52-e5c5-4235-b27b-197053c15c7d","Type":"ContainerStarted","Data":"1f8b92e36c475789649ef7010828cd4d90f3a0ee40623faf2f892713eacfc5fa"}
Sep 16 12:08:51.892761 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.892725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:51.996961 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:51.996883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:52.139708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:52.139672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:53.547027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:53.546958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:54.495852 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:54.495812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:54.599124 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:54.599094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:54.900391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:54.900353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:54.997383 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:54.997352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:55.294369 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:55.294284    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:55.407252 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:55.407208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:55.780484 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:55.780444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft" event={"ID":"b5b9ca52-e5c5-4235-b27b-197053c15c7d","Type":"ContainerStarted","Data":"4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9"}
Sep 16 12:08:55.780892 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:55.780546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:08:55.800662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:55.800613    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft" podStartSLOduration=1.529549883 podStartE2EDuration="4.800597321s" totalImagesPullingTime="3.271047438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:55.799734068 +0000 UTC m=+423.211750967" watchObservedRunningTime="2026-09-16 12:08:55.800597321 +0000 UTC m=+423.212614213"
Sep 16 12:08:55.962661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:55.962625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:58.432952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.432915    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"]
Sep 16 12:08:58.436269 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.436244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:58.439985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.439962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ktrdp\""
Sep 16 12:08:58.457746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.457723    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"]
Sep 16 12:08:58.476687 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.476665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjfv\" (UniqueName: \"kubernetes.io/projected/8603435e-57cc-4a0c-a297-3b337fbc487e-kube-api-access-gsjfv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:58.476795 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.476715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8603435e-57cc-4a0c-a297-3b337fbc487e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:58.577298 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.577271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjfv\" (UniqueName: \"kubernetes.io/projected/8603435e-57cc-4a0c-a297-3b337fbc487e-kube-api-access-gsjfv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:58.577435 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.577307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8603435e-57cc-4a0c-a297-3b337fbc487e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:58.577657 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.577641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8603435e-57cc-4a0c-a297-3b337fbc487e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:58.586774 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.586752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjfv\" (UniqueName: \"kubernetes.io/projected/8603435e-57cc-4a0c-a297-3b337fbc487e-kube-api-access-gsjfv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:58.746123 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:58.746030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:08:59.021341 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:59.021315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"]
Sep 16 12:08:59.022764 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:08:59.022735    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8603435e_57cc_4a0c_a297_3b337fbc487e.slice/crio-b4259072a81de7ef1d2c19f26e73f66964883cf994f057a0148ecb180898dcd3 WatchSource:0}: Error finding container b4259072a81de7ef1d2c19f26e73f66964883cf994f057a0148ecb180898dcd3: Status 404 returned error can't find the container with id b4259072a81de7ef1d2c19f26e73f66964883cf994f057a0148ecb180898dcd3
Sep 16 12:08:59.726777 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:59.726740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:08:59.800665 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:59.800633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq" event={"ID":"8603435e-57cc-4a0c-a297-3b337fbc487e","Type":"ContainerStarted","Data":"b4259072a81de7ef1d2c19f26e73f66964883cf994f057a0148ecb180898dcd3"}
Sep 16 12:08:59.821018 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:08:59.820965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:03.095453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:03.095431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:03.197473 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:03.197436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:03.822132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:03.822096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq" event={"ID":"8603435e-57cc-4a0c-a297-3b337fbc487e","Type":"ContainerStarted","Data":"2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0"}
Sep 16 12:09:03.822311 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:03.822211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:09:03.844784 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:03.844726    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq" podStartSLOduration=1.796386487 podStartE2EDuration="5.844712224s" totalImagesPullingTime="4.048325737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:03.842058002 +0000 UTC m=+431.254074898" watchObservedRunningTime="2026-09-16 12:09:03.844712224 +0000 UTC m=+431.256729120"
Sep 16 12:09:05.782139 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:05.782096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:05.914897 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:05.914866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:06.787041 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:06.787012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:07.560420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:07.560388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:07.686336 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:07.686301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:07.777152 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:07.777115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:12.238396 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:12.238358    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:12.336275 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:12.336240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:12.436013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:12.435980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:14.829735 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:14.829707    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:09:17.075343 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.075261    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"]
Sep 16 12:09:17.075750 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.075460    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq" podUID="8603435e-57cc-4a0c-a297-3b337fbc487e" containerName="manager" containerID="cri-o://2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0" gracePeriod=2
Sep 16 12:09:17.084453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.084317    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"]
Sep 16 12:09:17.087171 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.087137    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"8603435e-57cc-4a0c-a297-3b337fbc487e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-vqblq\": pods \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:09:17.099751 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.099716    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"]
Sep 16 12:09:17.100072 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.100056    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8603435e-57cc-4a0c-a297-3b337fbc487e" containerName="manager"
Sep 16 12:09:17.100145 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.100074    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8603435e-57cc-4a0c-a297-3b337fbc487e" containerName="manager"
Sep 16 12:09:17.100229 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.100200    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8603435e-57cc-4a0c-a297-3b337fbc487e" containerName="manager"
Sep 16 12:09:17.103099 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.103081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.106339 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.106307    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-vqblq\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="8603435e-57cc-4a0c-a297-3b337fbc487e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:09:17.117915 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.117879    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"]
Sep 16 12:09:17.118136 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.118107    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" containerName="manager" containerID="cri-o://4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9" gracePeriod=2
Sep 16 12:09:17.121897 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.121872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"]
Sep 16 12:09:17.135942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.135914    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"]
Sep 16 12:09:17.147352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.147326    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"]
Sep 16 12:09:17.147658 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.147644    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" containerName="manager"
Sep 16 12:09:17.147709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.147659    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" containerName="manager"
Sep 16 12:09:17.147744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.147730    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" containerName="manager"
Sep 16 12:09:17.150896 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.150879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"
Sep 16 12:09:17.154790 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.154753    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.168948 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.168925    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"]
Sep 16 12:09:17.233862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.233833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjsmx\" (UniqueName: \"kubernetes.io/projected/f6370efc-3bc9-4820-9f7a-a636521d1c01-kube-api-access-bjsmx\") pod \"limitador-operator-controller-manager-74495c69f8-wppcw\" (UID: \"f6370efc-3bc9-4820-9f7a-a636521d1c01\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"
Sep 16 12:09:17.233982 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.233884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f91c0e3-103c-43a6-bfec-735a9e01b04e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tlf52\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.233982 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.233909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5wt8\" (UniqueName: \"kubernetes.io/projected/0f91c0e3-103c-43a6-bfec-735a9e01b04e-kube-api-access-m5wt8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tlf52\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.335316 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.335240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f91c0e3-103c-43a6-bfec-735a9e01b04e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tlf52\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.335316 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.335279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5wt8\" (UniqueName: \"kubernetes.io/projected/0f91c0e3-103c-43a6-bfec-735a9e01b04e-kube-api-access-m5wt8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tlf52\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.335488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.335340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjsmx\" (UniqueName: \"kubernetes.io/projected/f6370efc-3bc9-4820-9f7a-a636521d1c01-kube-api-access-bjsmx\") pod \"limitador-operator-controller-manager-74495c69f8-wppcw\" (UID: \"f6370efc-3bc9-4820-9f7a-a636521d1c01\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"
Sep 16 12:09:17.335755 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.335728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f91c0e3-103c-43a6-bfec-735a9e01b04e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tlf52\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.344626 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.344558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjsmx\" (UniqueName: \"kubernetes.io/projected/f6370efc-3bc9-4820-9f7a-a636521d1c01-kube-api-access-bjsmx\") pod \"limitador-operator-controller-manager-74495c69f8-wppcw\" (UID: \"f6370efc-3bc9-4820-9f7a-a636521d1c01\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"
Sep 16 12:09:17.344626 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.344611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5wt8\" (UniqueName: \"kubernetes.io/projected/0f91c0e3-103c-43a6-bfec-735a9e01b04e-kube-api-access-m5wt8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tlf52\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.363262 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.363238    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:09:17.365013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.364989    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.365802 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.365780    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.367933 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.367911    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.436394 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.436359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/b5b9ca52-e5c5-4235-b27b-197053c15c7d-kube-api-access-rwdv9\") pod \"b5b9ca52-e5c5-4235-b27b-197053c15c7d\" (UID: \"b5b9ca52-e5c5-4235-b27b-197053c15c7d\") "
Sep 16 12:09:17.436563 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.436424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8603435e-57cc-4a0c-a297-3b337fbc487e-extensions-socket-volume\") pod \"8603435e-57cc-4a0c-a297-3b337fbc487e\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") "
Sep 16 12:09:17.436563 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.436456    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsjfv\" (UniqueName: \"kubernetes.io/projected/8603435e-57cc-4a0c-a297-3b337fbc487e-kube-api-access-gsjfv\") pod \"8603435e-57cc-4a0c-a297-3b337fbc487e\" (UID: \"8603435e-57cc-4a0c-a297-3b337fbc487e\") "
Sep 16 12:09:17.437004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.436974    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603435e-57cc-4a0c-a297-3b337fbc487e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8603435e-57cc-4a0c-a297-3b337fbc487e" (UID: "8603435e-57cc-4a0c-a297-3b337fbc487e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:09:17.438579 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.438543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8603435e-57cc-4a0c-a297-3b337fbc487e-kube-api-access-gsjfv" (OuterVolumeSpecName: "kube-api-access-gsjfv") pod "8603435e-57cc-4a0c-a297-3b337fbc487e" (UID: "8603435e-57cc-4a0c-a297-3b337fbc487e"). InnerVolumeSpecName "kube-api-access-gsjfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:17.438647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.438589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b9ca52-e5c5-4235-b27b-197053c15c7d-kube-api-access-rwdv9" (OuterVolumeSpecName: "kube-api-access-rwdv9") pod "b5b9ca52-e5c5-4235-b27b-197053c15c7d" (UID: "b5b9ca52-e5c5-4235-b27b-197053c15c7d"). InnerVolumeSpecName "kube-api-access-rwdv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:17.491108 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.491067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:17.497494 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.497472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"
Sep 16 12:09:17.537383 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.537342    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/b5b9ca52-e5c5-4235-b27b-197053c15c7d-kube-api-access-rwdv9\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:09:17.537383 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.537382    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8603435e-57cc-4a0c-a297-3b337fbc487e-extensions-socket-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:09:17.537668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.537399    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjfv\" (UniqueName: \"kubernetes.io/projected/8603435e-57cc-4a0c-a297-3b337fbc487e-kube-api-access-gsjfv\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:09:17.768794 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.768760    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"]
Sep 16 12:09:17.772220 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:09:17.772193    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f91c0e3_103c_43a6_bfec_735a9e01b04e.slice/crio-664070de96952e3d0d6ea46774bcf09b12f7407396bdf6ba7102c992a05d9fdf WatchSource:0}: Error finding container 664070de96952e3d0d6ea46774bcf09b12f7407396bdf6ba7102c992a05d9fdf: Status 404 returned error can't find the container with id 664070de96952e3d0d6ea46774bcf09b12f7407396bdf6ba7102c992a05d9fdf
Sep 16 12:09:17.790324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.790302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"]
Sep 16 12:09:17.792188 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:09:17.792165    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6370efc_3bc9_4820_9f7a_a636521d1c01.slice/crio-59119315bdcbace93bcc5c3da5c56cc4f3655d696cc1e793b2b466305f8a1c71 WatchSource:0}: Error finding container 59119315bdcbace93bcc5c3da5c56cc4f3655d696cc1e793b2b466305f8a1c71: Status 404 returned error can't find the container with id 59119315bdcbace93bcc5c3da5c56cc4f3655d696cc1e793b2b466305f8a1c71
Sep 16 12:09:17.886002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.885972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52" event={"ID":"0f91c0e3-103c-43a6-bfec-735a9e01b04e","Type":"ContainerStarted","Data":"664070de96952e3d0d6ea46774bcf09b12f7407396bdf6ba7102c992a05d9fdf"}
Sep 16 12:09:17.887860 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.887834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw" event={"ID":"f6370efc-3bc9-4820-9f7a-a636521d1c01","Type":"ContainerStarted","Data":"59119315bdcbace93bcc5c3da5c56cc4f3655d696cc1e793b2b466305f8a1c71"}
Sep 16 12:09:17.889145 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.889122    2577 generic.go:356] "Generic (PLEG): container finished" podID="8603435e-57cc-4a0c-a297-3b337fbc487e" containerID="2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0" exitCode=0
Sep 16 12:09:17.889232 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.889187    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqblq"
Sep 16 12:09:17.889294 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.889229    2577 scope.go:117] "RemoveContainer" containerID="2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0"
Sep 16 12:09:17.890736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.890705    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" containerID="4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9" exitCode=0
Sep 16 12:09:17.890825 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.890743    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.891760 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.891731    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.893946 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.893919    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.898660 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.898639    2577 scope.go:117] "RemoveContainer" containerID="2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0"
Sep 16 12:09:17.899002 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.898983    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0\": container with ID starting with 2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0 not found: ID does not exist" containerID="2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0"
Sep 16 12:09:17.899087 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.899007    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0"} err="failed to get container status \"2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0\": rpc error: code = NotFound desc = could not find container \"2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0\": container with ID starting with 2e9dec5cbf9bee2c1916e5eaa7cd721ba43b58a5dcbcef7c06aa3147c9c272f0 not found: ID does not exist"
Sep 16 12:09:17.899087 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.899024    2577 scope.go:117] "RemoveContainer" containerID="4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9"
Sep 16 12:09:17.904045 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.904019    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.906009 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.905986    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:17.906650 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.906636    2577 scope.go:117] "RemoveContainer" containerID="4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9"
Sep 16 12:09:17.906920 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:17.906900    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9\": container with ID starting with 4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9 not found: ID does not exist" containerID="4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9"
Sep 16 12:09:17.906954 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:17.906927    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9"} err="failed to get container status \"4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9\": rpc error: code = NotFound desc = could not find container \"4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9\": container with ID starting with 4e3e7a98a25a0d52b229c6cf3fb1ec195b9a9b0df9438dd56df9116a90db69a9 not found: ID does not exist"
Sep 16 12:09:18.648713 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.648682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:18.744010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.743979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:18.897614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.897562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52" event={"ID":"0f91c0e3-103c-43a6-bfec-735a9e01b04e","Type":"ContainerStarted","Data":"ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6"}
Sep 16 12:09:18.897800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.897671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:18.899231 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.899160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw" event={"ID":"f6370efc-3bc9-4820-9f7a-a636521d1c01","Type":"ContainerStarted","Data":"52c244685bc60eb62fb817930435079cfc1b51fb81c658f75891021ec7fb864b"}
Sep 16 12:09:18.899231 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.899222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"
Sep 16 12:09:18.922651 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:18.922554    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:18.922929 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.922893    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52" podStartSLOduration=1.922879601 podStartE2EDuration="1.922879601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:18.920129169 +0000 UTC m=+446.332146065" watchObservedRunningTime="2026-09-16 12:09:18.922879601 +0000 UTC m=+446.334896497"
Sep 16 12:09:18.924886 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:18.924859    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-vnrft\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vnrft"
Sep 16 12:09:18.946010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.945963    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw" podStartSLOduration=1.945949457 podStartE2EDuration="1.945949457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:18.945066169 +0000 UTC m=+446.357083066" watchObservedRunningTime="2026-09-16 12:09:18.945949457 +0000 UTC m=+446.357966354"
Sep 16 12:09:18.951414 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:18.951391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:19.048725 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:19.048692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:19.112443 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:19.112403    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8603435e-57cc-4a0c-a297-3b337fbc487e" path="/var/lib/kubelet/pods/8603435e-57cc-4a0c-a297-3b337fbc487e/volumes"
Sep 16 12:09:19.112759 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:19.112744    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b9ca52-e5c5-4235-b27b-197053c15c7d" path="/var/lib/kubelet/pods/b5b9ca52-e5c5-4235-b27b-197053c15c7d/volumes"
Sep 16 12:09:22.102546 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:22.102497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:22.203534 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:22.203506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:26.516208 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:26.516172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:26.617953 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:26.617915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:29.908016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:29.907988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-wppcw"
Sep 16 12:09:29.908462 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:29.908051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:31.263965 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:31.263929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:31.368136 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:31.368105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:31.437607 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:31.437559    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:32.555563 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.555530    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"]
Sep 16 12:09:32.555950 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.555782    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52" podUID="0f91c0e3-103c-43a6-bfec-735a9e01b04e" containerName="manager" containerID="cri-o://ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6" gracePeriod=10
Sep 16 12:09:32.805199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.805171    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:32.869395 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.869324    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"]
Sep 16 12:09:32.869659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.869644    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f91c0e3-103c-43a6-bfec-735a9e01b04e" containerName="manager"
Sep 16 12:09:32.869709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.869663    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f91c0e3-103c-43a6-bfec-735a9e01b04e" containerName="manager"
Sep 16 12:09:32.869743 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.869714    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f91c0e3-103c-43a6-bfec-735a9e01b04e" containerName="manager"
Sep 16 12:09:32.871466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.871450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:32.889357 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.889330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"]
Sep 16 12:09:32.960590 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.960544    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f91c0e3-103c-43a6-bfec-735a9e01b04e" containerID="ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6" exitCode=0
Sep 16 12:09:32.960746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.960608    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"
Sep 16 12:09:32.960746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.960632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52" event={"ID":"0f91c0e3-103c-43a6-bfec-735a9e01b04e","Type":"ContainerDied","Data":"ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6"}
Sep 16 12:09:32.960746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.960682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52" event={"ID":"0f91c0e3-103c-43a6-bfec-735a9e01b04e","Type":"ContainerDied","Data":"664070de96952e3d0d6ea46774bcf09b12f7407396bdf6ba7102c992a05d9fdf"}
Sep 16 12:09:32.960746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.960709    2577 scope.go:117] "RemoveContainer" containerID="ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6"
Sep 16 12:09:32.965592 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.965553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5wt8\" (UniqueName: \"kubernetes.io/projected/0f91c0e3-103c-43a6-bfec-735a9e01b04e-kube-api-access-m5wt8\") pod \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") "
Sep 16 12:09:32.965692 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.965645    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f91c0e3-103c-43a6-bfec-735a9e01b04e-extensions-socket-volume\") pod \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\" (UID: \"0f91c0e3-103c-43a6-bfec-735a9e01b04e\") "
Sep 16 12:09:32.965782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.965763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae2e741-85e7-4c95-9a78-0da2d617e8a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wlkdc\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:32.965910 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.965889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tn8\" (UniqueName: \"kubernetes.io/projected/aae2e741-85e7-4c95-9a78-0da2d617e8a9-kube-api-access-p4tn8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wlkdc\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:32.966021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.966001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f91c0e3-103c-43a6-bfec-735a9e01b04e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0f91c0e3-103c-43a6-bfec-735a9e01b04e" (UID: "0f91c0e3-103c-43a6-bfec-735a9e01b04e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:09:32.967653 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.967630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f91c0e3-103c-43a6-bfec-735a9e01b04e-kube-api-access-m5wt8" (OuterVolumeSpecName: "kube-api-access-m5wt8") pod "0f91c0e3-103c-43a6-bfec-735a9e01b04e" (UID: "0f91c0e3-103c-43a6-bfec-735a9e01b04e"). InnerVolumeSpecName "kube-api-access-m5wt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:32.968652 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.968635    2577 scope.go:117] "RemoveContainer" containerID="ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6"
Sep 16 12:09:32.968932 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:09:32.968912    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6\": container with ID starting with ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6 not found: ID does not exist" containerID="ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6"
Sep 16 12:09:32.968984 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:32.968941    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6"} err="failed to get container status \"ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6\": rpc error: code = NotFound desc = could not find container \"ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6\": container with ID starting with ae173256f956e7d8693983d8dc8086ab410514d4916cc4207577a234e444bbe6 not found: ID does not exist"
Sep 16 12:09:33.066680 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.066642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae2e741-85e7-4c95-9a78-0da2d617e8a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wlkdc\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:33.066862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.066709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tn8\" (UniqueName: \"kubernetes.io/projected/aae2e741-85e7-4c95-9a78-0da2d617e8a9-kube-api-access-p4tn8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wlkdc\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:33.066862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.066758    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5wt8\" (UniqueName: \"kubernetes.io/projected/0f91c0e3-103c-43a6-bfec-735a9e01b04e-kube-api-access-m5wt8\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:09:33.066862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.066768    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f91c0e3-103c-43a6-bfec-735a9e01b04e-extensions-socket-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:09:33.067050 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.067029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae2e741-85e7-4c95-9a78-0da2d617e8a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wlkdc\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:33.075528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.075495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tn8\" (UniqueName: \"kubernetes.io/projected/aae2e741-85e7-4c95-9a78-0da2d617e8a9-kube-api-access-p4tn8\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wlkdc\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:33.181420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.181387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:33.277361 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.277334    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"]
Sep 16 12:09:33.284237 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.284213    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tlf52"]
Sep 16 12:09:33.456288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.456214    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"]
Sep 16 12:09:33.459678 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:09:33.459634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae2e741_85e7_4c95_9a78_0da2d617e8a9.slice/crio-0ecd9e1314118e5be2c40ba2e8c43f5d36987f5ae5955dfdc792ea189b13fb0c WatchSource:0}: Error finding container 0ecd9e1314118e5be2c40ba2e8c43f5d36987f5ae5955dfdc792ea189b13fb0c: Status 404 returned error can't find the container with id 0ecd9e1314118e5be2c40ba2e8c43f5d36987f5ae5955dfdc792ea189b13fb0c
Sep 16 12:09:33.966590 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.966532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc" event={"ID":"aae2e741-85e7-4c95-9a78-0da2d617e8a9","Type":"ContainerStarted","Data":"56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a"}
Sep 16 12:09:33.966995 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.966734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:33.966995 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.966782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc" event={"ID":"aae2e741-85e7-4c95-9a78-0da2d617e8a9","Type":"ContainerStarted","Data":"0ecd9e1314118e5be2c40ba2e8c43f5d36987f5ae5955dfdc792ea189b13fb0c"}
Sep 16 12:09:33.990071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:33.990025    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc" podStartSLOduration=1.990011151 podStartE2EDuration="1.990011151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:33.988015308 +0000 UTC m=+461.400032201" watchObservedRunningTime="2026-09-16 12:09:33.990011151 +0000 UTC m=+461.402028046"
Sep 16 12:09:35.110691 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:35.110656    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f91c0e3-103c-43a6-bfec-735a9e01b04e" path="/var/lib/kubelet/pods/0f91c0e3-103c-43a6-bfec-735a9e01b04e/volumes"
Sep 16 12:09:44.974466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:44.974431    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:09:48.988113 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:48.988076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:09:49.245500 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:09:49.245412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:01.653949 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:01.653920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:01.708808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:01.708772    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:02.966626 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:02.966552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:03.069828 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.069796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:03.172145 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.172116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:03.269291 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.269191    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:10:03.273702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.273678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.276593 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.276556    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4k8h2\""
Sep 16 12:10:03.276711 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.276633    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 12:10:03.283533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.283511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:10:03.312366 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.312330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:10:03.406512 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.406471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ea5973c9-115f-4a33-bedb-933f2b48e239-config-file\") pod \"limitador-limitador-84994f7ffc-dqgg6\" (UID: \"ea5973c9-115f-4a33-bedb-933f2b48e239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.406512 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.406513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbsm\" (UniqueName: \"kubernetes.io/projected/ea5973c9-115f-4a33-bedb-933f2b48e239-kube-api-access-6cbsm\") pod \"limitador-limitador-84994f7ffc-dqgg6\" (UID: \"ea5973c9-115f-4a33-bedb-933f2b48e239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.507591 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.507532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ea5973c9-115f-4a33-bedb-933f2b48e239-config-file\") pod \"limitador-limitador-84994f7ffc-dqgg6\" (UID: \"ea5973c9-115f-4a33-bedb-933f2b48e239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.507754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.507589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbsm\" (UniqueName: \"kubernetes.io/projected/ea5973c9-115f-4a33-bedb-933f2b48e239-kube-api-access-6cbsm\") pod \"limitador-limitador-84994f7ffc-dqgg6\" (UID: \"ea5973c9-115f-4a33-bedb-933f2b48e239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.508165 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.508145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ea5973c9-115f-4a33-bedb-933f2b48e239-config-file\") pod \"limitador-limitador-84994f7ffc-dqgg6\" (UID: \"ea5973c9-115f-4a33-bedb-933f2b48e239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.516400 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.516380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbsm\" (UniqueName: \"kubernetes.io/projected/ea5973c9-115f-4a33-bedb-933f2b48e239-kube-api-access-6cbsm\") pod \"limitador-limitador-84994f7ffc-dqgg6\" (UID: \"ea5973c9-115f-4a33-bedb-933f2b48e239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.584113 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.584034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:03.785938 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.785904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:03.849244 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.849163    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:10:03.851862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:03.851838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:03.852970 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:10:03.852948    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5973c9_115f_4a33_bedb_933f2b48e239.slice/crio-5f9b88213ab99c4c265a571810510f9592578147640ebac3d83ee0392a32edf4 WatchSource:0}: Error finding container 5f9b88213ab99c4c265a571810510f9592578147640ebac3d83ee0392a32edf4: Status 404 returned error can't find the container with id 5f9b88213ab99c4c265a571810510f9592578147640ebac3d83ee0392a32edf4
Sep 16 12:10:04.102387 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:04.102308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6" event={"ID":"ea5973c9-115f-4a33-bedb-933f2b48e239","Type":"ContainerStarted","Data":"5f9b88213ab99c4c265a571810510f9592578147640ebac3d83ee0392a32edf4"}
Sep 16 12:10:05.110527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:05.110491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6" event={"ID":"ea5973c9-115f-4a33-bedb-933f2b48e239","Type":"ContainerStarted","Data":"9f38287ac2c8ccb11ef88de4bf36fb8819aae99535592c0857bc31121b8c34a5"}
Sep 16 12:10:05.110916 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:05.110556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:05.125539 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:05.125484    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6" podStartSLOduration=0.998417563 podStartE2EDuration="2.125468929s" totalImagesPullingTime="1.127051366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:05.124073589 +0000 UTC m=+492.536090480" watchObservedRunningTime="2026-09-16 12:10:05.125468929 +0000 UTC m=+492.537485826"
Sep 16 12:10:16.113004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:16.112974    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"
Sep 16 12:10:18.109891 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:18.109847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:18.207330 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:18.207283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:18.665176 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:18.665137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:31.144367 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:31.144330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:31.572849 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:31.572767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:38.300092 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:38.300059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:38.400891 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:38.400856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:38.492689 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:38.492663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:38.805266 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:38.805232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:42.239230 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:42.239195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:10:42.338687 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:10:42.338646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:05.634972 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:05.634940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:08.832070 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:08.832037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:08.889080 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:08.889049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:14.682118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:14.682084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:14.775145 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:14.775112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:16.152484 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:16.152454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:16.230424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:16.230379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:16.312547 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:16.312512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:16.440253 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:16.440172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:16.553287 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:16.553252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:16.654339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:16.654304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:16.708392 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:16.708310    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:11:25.754944 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:25.754902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:25.860292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:25.860258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:39.377854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:39.377819    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:39.993033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:39.992995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:11:47.973939 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:47.973903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:48.688533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:48.688478    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:11:55.582819 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:55.582781    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:11:56.482363 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:11:56.482325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:12:03.644253 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.644217    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"]
Sep 16 12:12:03.647347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.647329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.650172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.650140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:12:03.650172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.650161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5j28l\""
Sep 16 12:12:03.651516 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.651493    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 12:12:03.651671 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.651654    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:12:03.656698 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.656675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"]
Sep 16 12:12:03.682707 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.682677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:03.706900 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.706871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.707033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.706906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.707033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.706925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.707033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.706949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.707170 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.707032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.707170 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.707079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqbs\" (UniqueName: \"kubernetes.io/projected/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-kube-api-access-glqbs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.808288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.808252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.808288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.808305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqbs\" (UniqueName: \"kubernetes.io/projected/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-kube-api-access-glqbs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.809531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.808405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.809531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.808438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.809531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.808464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.809531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.808492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.809531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.808786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.809531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.809029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.809531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.809115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.811104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.811079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.811308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.811293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.816496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.816474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqbs\" (UniqueName: \"kubernetes.io/projected/a514860c-5bb5-4d0f-94c2-b1b45f5a3efb-kube-api-access-glqbs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6q84w\" (UID: \"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:03.958461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:03.958371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:04.249646 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:04.249594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"]
Sep 16 12:12:04.251989 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:12:04.251956    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda514860c_5bb5_4d0f_94c2_b1b45f5a3efb.slice/crio-da9883864b3b4ca72838e810e97e339dde7cbe1136e0e2bed08d3a928dae8759 WatchSource:0}: Error finding container da9883864b3b4ca72838e810e97e339dde7cbe1136e0e2bed08d3a928dae8759: Status 404 returned error can't find the container with id da9883864b3b4ca72838e810e97e339dde7cbe1136e0e2bed08d3a928dae8759
Sep 16 12:12:04.632307 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:04.632270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w" event={"ID":"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb","Type":"ContainerStarted","Data":"da9883864b3b4ca72838e810e97e339dde7cbe1136e0e2bed08d3a928dae8759"}
Sep 16 12:12:05.112949 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:05.112871    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:12:05.638271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:05.638231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w" event={"ID":"a514860c-5bb5-4d0f-94c2-b1b45f5a3efb","Type":"ContainerStarted","Data":"5d8cb687cdb59c764c72afa65581bc8100e6ffe5c18daa56c6a5962c71781480"}
Sep 16 12:12:05.664790 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:05.664727    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w" podStartSLOduration=1.685874297 podStartE2EDuration="2.664705807s" totalImagesPullingTime="978.83151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:05.663750507 +0000 UTC m=+613.075767419" watchObservedRunningTime="2026-09-16 12:12:05.664705807 +0000 UTC m=+613.076722705"
Sep 16 12:12:13.959307 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:13.959268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:13.959814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:13.959426    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:13.971812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:13.971778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:14.466350 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:14.466309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:14.687270 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:14.687243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6q84w"
Sep 16 12:12:15.985377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:15.985295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:12:30.344463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.344432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"]
Sep 16 12:12:30.348012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.347990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.350783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.350765    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 12:12:30.357893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.357871    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"]
Sep 16 12:12:30.387414 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.387383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:30.440899 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.440813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.441045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.440911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b471c6-472e-480f-9617-a6ebaa6cf1dc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.441045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.440960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.441045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.440989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.441045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.441014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzlw9\" (UniqueName: \"kubernetes.io/projected/65b471c6-472e-480f-9617-a6ebaa6cf1dc-kube-api-access-xzlw9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.441193 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.441096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.541661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.541621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b471c6-472e-480f-9617-a6ebaa6cf1dc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.541843 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.541675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.541843 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.541710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.541843 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.541734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzlw9\" (UniqueName: \"kubernetes.io/projected/65b471c6-472e-480f-9617-a6ebaa6cf1dc-kube-api-access-xzlw9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.541843 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.541801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.542071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.541865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.542175 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.542143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.542263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.542199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.542334 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.542305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.544175 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.544145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b471c6-472e-480f-9617-a6ebaa6cf1dc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.544290 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.544274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b471c6-472e-480f-9617-a6ebaa6cf1dc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.549684 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.549662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzlw9\" (UniqueName: \"kubernetes.io/projected/65b471c6-472e-480f-9617-a6ebaa6cf1dc-kube-api-access-xzlw9\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-plrcp\" (UID: \"65b471c6-472e-480f-9617-a6ebaa6cf1dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.658895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.658851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:30.929994 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.929966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"]
Sep 16 12:12:30.932215 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:12:30.932182    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b471c6_472e_480f_9617_a6ebaa6cf1dc.slice/crio-ac0a00c7fd95ce341b4f19596c17c19e0c9afc29084496095d2559d6027681de WatchSource:0}: Error finding container ac0a00c7fd95ce341b4f19596c17c19e0c9afc29084496095d2559d6027681de: Status 404 returned error can't find the container with id ac0a00c7fd95ce341b4f19596c17c19e0c9afc29084496095d2559d6027681de
Sep 16 12:12:30.933904 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:30.933879    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:12:31.750385 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:31.750341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp" event={"ID":"65b471c6-472e-480f-9617-a6ebaa6cf1dc","Type":"ContainerStarted","Data":"bc5cc546a4aed5d2342fd62432136ba40f135c0668975ce593472cdf5d60b76d"}
Sep 16 12:12:31.750835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:31.750412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp" event={"ID":"65b471c6-472e-480f-9617-a6ebaa6cf1dc","Type":"ContainerStarted","Data":"ac0a00c7fd95ce341b4f19596c17c19e0c9afc29084496095d2559d6027681de"}
Sep 16 12:12:31.769375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:31.769315    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp" podStartSLOduration=1.132813732 podStartE2EDuration="1.769297579s" totalImagesPullingTime="636.483847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:31.768031159 +0000 UTC m=+639.180048055" watchObservedRunningTime="2026-09-16 12:12:31.769297579 +0000 UTC m=+639.181314474"
Sep 16 12:12:32.286399 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:32.286346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:12:40.659313 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:40.659268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:40.659775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:40.659359    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:40.671892 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:40.671864    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:40.801322 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:40.801292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-plrcp"
Sep 16 12:12:44.987087 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:44.987049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:12:47.180050 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:12:47.180007    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:13:18.749106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:18.749052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:18.849332 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:18.849301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:27.737498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:27.737455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:27.790327 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:27.790274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:44.073557 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:44.073525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:44.173600 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:44.173555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:45.382633 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:45.382521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:13:45.479747 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:13:45.479711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:05.922471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:05.922432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:05.984930 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:05.984895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:06.058616 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:06.058587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:06.349760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:06.349678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:06.453595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:06.453540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:06.555032 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:06.555002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:06.652051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:06.651953    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:07.216405 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:07.216370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:07.496492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:07.496403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:07.613395 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:07.613353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:07.948404 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:07.948372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:09.798485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:09.798448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:14:09.988757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:09.988728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:10.125014 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:10.124920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:10.244237 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:10.244198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:11.622066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:11.622037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:11.720178 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:11.720145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:11.979319 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:11.979289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:12.144798 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.144753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:12.245858 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.245763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:12.283338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.283292    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"]
Sep 16 12:14:12.286838 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.286814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.290749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.290724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-641983-dockercfg-pnv5l\""
Sep 16 12:14:12.291012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.290996    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-641983\""
Sep 16 12:14:12.297856 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.297833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"]
Sep 16 12:14:12.420727 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.420683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c855a2-9e46-448a-b01f-1cc4205de1c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.420902 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.420740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbsxv\" (UniqueName: \"kubernetes.io/projected/82c855a2-9e46-448a-b01f-1cc4205de1c4-kube-api-access-rbsxv\") pod \"payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.521662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.521550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c855a2-9e46-448a-b01f-1cc4205de1c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.521662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.521638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbsxv\" (UniqueName: \"kubernetes.io/projected/82c855a2-9e46-448a-b01f-1cc4205de1c4-kube-api-access-rbsxv\") pod \"payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.522286 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.522265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c855a2-9e46-448a-b01f-1cc4205de1c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.529543 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.529511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbsxv\" (UniqueName: \"kubernetes.io/projected/82c855a2-9e46-448a-b01f-1cc4205de1c4-kube-api-access-rbsxv\") pod \"payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.597546 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.597507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:12.675106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.675047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:12.766790 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.766754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:12.858371 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.858289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:12.897445 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:12.897415    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"]
Sep 16 12:14:12.902604 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:12.902549    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c855a2_9e46_448a_b01f_1cc4205de1c4.slice/crio-f53c540425c54d35be7027cb5c63ed6c66b19ce746e3c5d2f5c0c004333813b2 WatchSource:0}: Error finding container f53c540425c54d35be7027cb5c63ed6c66b19ce746e3c5d2f5c0c004333813b2: Status 404 returned error can't find the container with id f53c540425c54d35be7027cb5c63ed6c66b19ce746e3c5d2f5c0c004333813b2
Sep 16 12:14:13.200594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.200541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw" event={"ID":"82c855a2-9e46-448a-b01f-1cc4205de1c4","Type":"ContainerStarted","Data":"f53c540425c54d35be7027cb5c63ed6c66b19ce746e3c5d2f5c0c004333813b2"}
Sep 16 12:14:13.282353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.282317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:13.425304 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.425272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:13.447892 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.447856    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"]
Sep 16 12:14:13.453591 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.453422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.457046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.457010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-556077-dockercfg-whlcg\""
Sep 16 12:14:13.457159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.457010    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-556077\""
Sep 16 12:14:13.465549 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.465524    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"]
Sep 16 12:14:13.634311 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.634274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6def81ab-d812-4022-b577-6dda69637422-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.634496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.634325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tn5x\" (UniqueName: \"kubernetes.io/projected/6def81ab-d812-4022-b577-6dda69637422-kube-api-access-4tn5x\") pod \"payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.702586 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.702532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:13.735434 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.735345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6def81ab-d812-4022-b577-6dda69637422-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.735434 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.735401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tn5x\" (UniqueName: \"kubernetes.io/projected/6def81ab-d812-4022-b577-6dda69637422-kube-api-access-4tn5x\") pod \"payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.736687 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.736633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6def81ab-d812-4022-b577-6dda69637422-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.745789 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.745759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tn5x\" (UniqueName: \"kubernetes.io/projected/6def81ab-d812-4022-b577-6dda69637422-kube-api-access-4tn5x\") pod \"payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.765588 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.765508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:13.803032 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.802989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:13.935254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:13.935211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:14.065438 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:14.065400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"]
Sep 16 12:14:14.261321 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:14.261275    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6def81ab_d812_4022_b577_6dda69637422.slice/crio-53441ad0c00c4187bb03fe0906e5afafc304dfa2f91d73a193eb70e050ee18a7 WatchSource:0}: Error finding container 53441ad0c00c4187bb03fe0906e5afafc304dfa2f91d73a193eb70e050ee18a7: Status 404 returned error can't find the container with id 53441ad0c00c4187bb03fe0906e5afafc304dfa2f91d73a193eb70e050ee18a7
Sep 16 12:14:14.320367 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:14.320324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:14.461684 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:14.461647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:14.836278 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:14.836242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:15.071760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.071701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:15.129472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.129439    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"]
Sep 16 12:14:15.133480 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.133456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.136037 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.136018    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-dc3e2c-openshift-default-dockercfg-t5l4w\""
Sep 16 12:14:15.145619 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.145596    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"]
Sep 16 12:14:15.178728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.178695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:15.214250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.214215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw" event={"ID":"82c855a2-9e46-448a-b01f-1cc4205de1c4","Type":"ContainerStarted","Data":"ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5"}
Sep 16 12:14:15.214907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.214850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:15.216591 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.216033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb" event={"ID":"6def81ab-d812-4022-b577-6dda69637422","Type":"ContainerStarted","Data":"a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b"}
Sep 16 12:14:15.216935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.216913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb" event={"ID":"6def81ab-d812-4022-b577-6dda69637422","Type":"ContainerStarted","Data":"53441ad0c00c4187bb03fe0906e5afafc304dfa2f91d73a193eb70e050ee18a7"}
Sep 16 12:14:15.217114 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.217099    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:15.237295 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.237226    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw" podStartSLOduration=1.835527556 podStartE2EDuration="3.237210054s" totalImagesPullingTime="1.401682498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:15.236360607 +0000 UTC m=+742.648377504" watchObservedRunningTime="2026-09-16 12:14:15.237210054 +0000 UTC m=+742.649226949"
Sep 16 12:14:15.247006 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.246971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619093be-c45c-48cd-a541-f6eb6df4323c-istio-podinfo\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-envoy\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-data\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619093be-c45c-48cd-a541-f6eb6df4323c-istiod-ca-cert\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-socket\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-istio-token\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nn7c\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-kube-api-access-5nn7c\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247443 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-certs\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.247443 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.247380    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-credential-socket\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.255376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.255328    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb" podStartSLOduration=1.900772199 podStartE2EDuration="2.255314592s" totalImagesPullingTime="354.542393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:15.254295334 +0000 UTC m=+742.666312280" watchObservedRunningTime="2026-09-16 12:14:15.255314592 +0000 UTC m=+742.667331488"
Sep 16 12:14:15.348173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-envoy\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-data\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348399 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619093be-c45c-48cd-a541-f6eb6df4323c-istiod-ca-cert\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348399 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-socket\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348522 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-istio-token\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348522 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nn7c\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-kube-api-access-5nn7c\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348642 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-certs\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348642 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-credential-socket\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348642 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619093be-c45c-48cd-a541-f6eb6df4323c-istio-podinfo\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348798 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-data\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.348798 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.348710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-socket\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.349045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.349016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-credential-socket\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.349109 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.349043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-certs\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.349109 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.349051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619093be-c45c-48cd-a541-f6eb6df4323c-istiod-ca-cert\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.350506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.350483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-envoy\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.350981 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.350963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619093be-c45c-48cd-a541-f6eb6df4323c-istio-podinfo\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.361215 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.361189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-istio-token\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.361392 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.361297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nn7c\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-kube-api-access-5nn7c\") pod \"e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") " pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.448636 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.448563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:15.629754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.629717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:15.740273 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.740206    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"]
Sep 16 12:14:15.742423 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:15.742394    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619093be_c45c_48cd_a541_f6eb6df4323c.slice/crio-cfbcbb450a15c46ad2b8ae12c055c6778ebf2ff64b240a4bbbe1b2a0300f3d8f WatchSource:0}: Error finding container cfbcbb450a15c46ad2b8ae12c055c6778ebf2ff64b240a4bbbe1b2a0300f3d8f: Status 404 returned error can't find the container with id cfbcbb450a15c46ad2b8ae12c055c6778ebf2ff64b240a4bbbe1b2a0300f3d8f
Sep 16 12:14:15.897918 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:15.897841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:16.222294 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:16.222186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" event={"ID":"619093be-c45c-48cd-a541-f6eb6df4323c","Type":"ContainerStarted","Data":"cfbcbb450a15c46ad2b8ae12c055c6778ebf2ff64b240a4bbbe1b2a0300f3d8f"}
Sep 16 12:14:16.329801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:16.329755    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:16.430828 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:16.430788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:16.562442 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:16.562360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:16.663541 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:16.663506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:16.960905 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:16.960841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:17.174273 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:17.174230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:18.327973 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:18.327936    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:14:18.328282 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:18.328015    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:14:18.328282 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:18.328045    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:14:19.240012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:19.239981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" event={"ID":"619093be-c45c-48cd-a541-f6eb6df4323c","Type":"ContainerStarted","Data":"6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4"}
Sep 16 12:14:19.265957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:19.265910    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podStartSLOduration=1.6827673079999999 podStartE2EDuration="4.265893798s" totalImagesPullingTime="2.58312649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:19.264523181 +0000 UTC m=+746.676540076" watchObservedRunningTime="2026-09-16 12:14:19.265893798 +0000 UTC m=+746.677910693"
Sep 16 12:14:19.449762 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:19.449724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:19.454913 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:19.454888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:20.245293 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:20.245259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:20.246261 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:20.246236    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:20.755343 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:20.755295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:21.258065 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:21.258012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:21.348990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:21.348952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:22.158345 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:22.158307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:22.292924 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:22.292873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:22.386860 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:22.386804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:22.532506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:22.532410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:23.928525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:23.928488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:24.771609 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:24.771549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:24.871685 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:24.871619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:25.123955 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:25.123849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:25.294159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:25.294126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:25.397154 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:25.397119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:25.489874 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:25.489839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:14:25.491216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:25.491191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:26.223332 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:26.223296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:26.312823 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:26.312794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:26.411670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:26.411625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:26.796527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:26.796494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:26.908563 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:26.908528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:27.090690 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.090612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:27.229769 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.229729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:14:27.230203 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.230159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:14:27.239246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.239216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:27.303106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.303073    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"]
Sep 16 12:14:27.306806 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.306779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.309476 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.309454    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67e03001\""
Sep 16 12:14:27.309655 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.309504    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67e03001-dockercfg-qhghf\""
Sep 16 12:14:27.320880 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.320845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"]
Sep 16 12:14:27.333584 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.333534    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"]
Sep 16 12:14:27.337360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.337336    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.354911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.354876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"]
Sep 16 12:14:27.356366 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.356333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9h4\" (UniqueName: \"kubernetes.io/projected/8580422c-9106-42e1-ad92-a7a9fef6c64c-kube-api-access-vx9h4\") pod \"payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.356468 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.356409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jh2\" (UniqueName: \"kubernetes.io/projected/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-kube-api-access-r9jh2\") pod \"payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") " pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.356468 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.356444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8580422c-9106-42e1-ad92-a7a9fef6c64c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.356629 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.356611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-plugins-config-volume\") pod \"payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") " pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.414949 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.414896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:27.457738 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.457707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-plugins-config-volume\") pod \"payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") " pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.457872 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.457744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9h4\" (UniqueName: \"kubernetes.io/projected/8580422c-9106-42e1-ad92-a7a9fef6c64c-kube-api-access-vx9h4\") pod \"payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.457872 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.457842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jh2\" (UniqueName: \"kubernetes.io/projected/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-kube-api-access-r9jh2\") pod \"payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") " pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.457872 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.457861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8580422c-9106-42e1-ad92-a7a9fef6c64c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.458527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.458506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8580422c-9106-42e1-ad92-a7a9fef6c64c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.458586 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.458510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-plugins-config-volume\") pod \"payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") " pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.474270 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.474236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jh2\" (UniqueName: \"kubernetes.io/projected/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-kube-api-access-r9jh2\") pod \"payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") " pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.474436 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.474247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9h4\" (UniqueName: \"kubernetes.io/projected/8580422c-9106-42e1-ad92-a7a9fef6c64c-kube-api-access-vx9h4\") pod \"payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.619537 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.619454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:27.653482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.653447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:27.897869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.897835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:27.898499 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.898477    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"]
Sep 16 12:14:27.900507 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:27.900484    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbef05b4_7b67_4b44_8175_ae1c05e48fd2.slice/crio-bb87f7f32679222289963e38530952d23faa71f37c8a085c79ef98f9355a1aca WatchSource:0}: Error finding container bb87f7f32679222289963e38530952d23faa71f37c8a085c79ef98f9355a1aca: Status 404 returned error can't find the container with id bb87f7f32679222289963e38530952d23faa71f37c8a085c79ef98f9355a1aca
Sep 16 12:14:27.936550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.936526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"]
Sep 16 12:14:27.938657 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:27.938635    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8580422c_9106_42e1_ad92_a7a9fef6c64c.slice/crio-35640131f7bfe5bec1101f0de9371773e629f2f6aa9bf39dfb004b63bd5b2639 WatchSource:0}: Error finding container 35640131f7bfe5bec1101f0de9371773e629f2f6aa9bf39dfb004b63bd5b2639: Status 404 returned error can't find the container with id 35640131f7bfe5bec1101f0de9371773e629f2f6aa9bf39dfb004b63bd5b2639
Sep 16 12:14:27.995203 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:27.995178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:28.249786 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.249705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:28.288126 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.288091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p" event={"ID":"8580422c-9106-42e1-ad92-a7a9fef6c64c","Type":"ContainerStarted","Data":"7e4a57e88997f09b43c5e8c488687cd332a84ec67c5cb0efda4641bc97687039"}
Sep 16 12:14:28.288336 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.288150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p" event={"ID":"8580422c-9106-42e1-ad92-a7a9fef6c64c","Type":"ContainerStarted","Data":"35640131f7bfe5bec1101f0de9371773e629f2f6aa9bf39dfb004b63bd5b2639"}
Sep 16 12:14:28.288336 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.288179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:28.289733 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.289710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm" event={"ID":"cbef05b4-7b67-4b44-8175-ae1c05e48fd2","Type":"ContainerStarted","Data":"4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924"}
Sep 16 12:14:28.289859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.289750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm" event={"ID":"cbef05b4-7b67-4b44-8175-ae1c05e48fd2","Type":"ContainerStarted","Data":"bb87f7f32679222289963e38530952d23faa71f37c8a085c79ef98f9355a1aca"}
Sep 16 12:14:28.289859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.289770    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:28.307342 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.307276    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p" podStartSLOduration=1.3072575180000001 podStartE2EDuration="1.307257518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:28.305324858 +0000 UTC m=+755.717341777" watchObservedRunningTime="2026-09-16 12:14:28.307257518 +0000 UTC m=+755.719274415"
Sep 16 12:14:28.328853 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.328549    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm" podStartSLOduration=1.32852314 podStartE2EDuration="1.32852314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:28.327952657 +0000 UTC m=+755.739969553" watchObservedRunningTime="2026-09-16 12:14:28.32852314 +0000 UTC m=+755.740540039"
Sep 16 12:14:28.350850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.350819    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:28.454711 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:28.454676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:29.071259 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:29.071219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:29.349147 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:29.349059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:29.433848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:29.433812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:29.789252 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:29.789215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:29.956382 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:29.956336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:30.070817 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:30.070740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:30.463618 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:30.463560    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:30.560434 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:30.560393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:30.661135 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:30.661105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:30.792387 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:30.792281    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:31.519839 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:31.519807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:31.719482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:31.719442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:31.818731 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:31.818649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:31.964459 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:31.964427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:32.581845 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:32.581812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:33.526496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:33.526459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:33.627488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:33.627454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:33.943661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:33.943612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:34.246528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.246433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:34.346385 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.346355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:34.443786 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.443750    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"]
Sep 16 12:14:34.449638 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.449614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.452441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.452420    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-08170a\""
Sep 16 12:14:34.457625 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.457601    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"]
Sep 16 12:14:34.524349 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.524262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2268e65e-0da8-4714-af81-385c06123bb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.524349 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.524314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fnwt\" (UniqueName: \"kubernetes.io/projected/2268e65e-0da8-4714-af81-385c06123bb1-kube-api-access-8fnwt\") pod \"payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.556736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.556698    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"]
Sep 16 12:14:34.560765 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.560743    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:34.563437 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.563398    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-08170a-dockercfg-whddq\""
Sep 16 12:14:34.570461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.570435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"]
Sep 16 12:14:34.624902 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.624873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bff3188-5f14-4c25-8534-1ae8b4c48b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:34.625065 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.624934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2268e65e-0da8-4714-af81-385c06123bb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.625065 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.624972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kj2b\" (UniqueName: \"kubernetes.io/projected/8bff3188-5f14-4c25-8534-1ae8b4c48b59-kube-api-access-2kj2b\") pod \"payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:34.625065 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.625038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fnwt\" (UniqueName: \"kubernetes.io/projected/2268e65e-0da8-4714-af81-385c06123bb1-kube-api-access-8fnwt\") pod \"payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.625718 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.625695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2268e65e-0da8-4714-af81-385c06123bb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.634167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.634140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fnwt\" (UniqueName: \"kubernetes.io/projected/2268e65e-0da8-4714-af81-385c06123bb1-kube-api-access-8fnwt\") pod \"payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.725951 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.725912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kj2b\" (UniqueName: \"kubernetes.io/projected/8bff3188-5f14-4c25-8534-1ae8b4c48b59-kube-api-access-2kj2b\") pod \"payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:34.726297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.726269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bff3188-5f14-4c25-8534-1ae8b4c48b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:34.727333 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.727308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bff3188-5f14-4c25-8534-1ae8b4c48b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:34.744591 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.742265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kj2b\" (UniqueName: \"kubernetes.io/projected/8bff3188-5f14-4c25-8534-1ae8b4c48b59-kube-api-access-2kj2b\") pod \"payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:34.764015 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.763980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:34.874289 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:34.874252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:35.047654 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.046941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"]
Sep 16 12:14:35.058818 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:35.058787    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2268e65e_0da8_4714_af81_385c06123bb1.slice/crio-648be7c830b0e6478500d05a2348a8c2aaf7884ee75395319f408e2da8566d57 WatchSource:0}: Error finding container 648be7c830b0e6478500d05a2348a8c2aaf7884ee75395319f408e2da8566d57: Status 404 returned error can't find the container with id 648be7c830b0e6478500d05a2348a8c2aaf7884ee75395319f408e2da8566d57
Sep 16 12:14:35.167028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.166997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"]
Sep 16 12:14:35.168454 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:35.168420    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bff3188_5f14_4c25_8534_1ae8b4c48b59.slice/crio-b8160be48ce0bf79521cc2515df062fb95ff18d80b61cdcc7f3e5b120424d382 WatchSource:0}: Error finding container b8160be48ce0bf79521cc2515df062fb95ff18d80b61cdcc7f3e5b120424d382: Status 404 returned error can't find the container with id b8160be48ce0bf79521cc2515df062fb95ff18d80b61cdcc7f3e5b120424d382
Sep 16 12:14:35.215727 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.215701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:35.324518 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.324437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs" event={"ID":"2268e65e-0da8-4714-af81-385c06123bb1","Type":"ContainerStarted","Data":"348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7"}
Sep 16 12:14:35.324518 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.324492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs" event={"ID":"2268e65e-0da8-4714-af81-385c06123bb1","Type":"ContainerStarted","Data":"648be7c830b0e6478500d05a2348a8c2aaf7884ee75395319f408e2da8566d57"}
Sep 16 12:14:35.324766 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.324551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:35.326587 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.326541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d" event={"ID":"8bff3188-5f14-4c25-8534-1ae8b4c48b59","Type":"ContainerStarted","Data":"acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4"}
Sep 16 12:14:35.326713 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.326613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d" event={"ID":"8bff3188-5f14-4c25-8534-1ae8b4c48b59","Type":"ContainerStarted","Data":"b8160be48ce0bf79521cc2515df062fb95ff18d80b61cdcc7f3e5b120424d382"}
Sep 16 12:14:35.327153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.327130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:35.345290 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:35.345222    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs" podStartSLOduration=1.345199251 podStartE2EDuration="1.345199251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:35.343182155 +0000 UTC m=+762.755199054" watchObservedRunningTime="2026-09-16 12:14:35.345199251 +0000 UTC m=+762.757216149"
Sep 16 12:14:36.279131 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:36.279096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:37.464122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:37.464088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:37.674017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:37.673986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:37.758173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:37.758052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:37.846884 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:37.846805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:38.097383 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:38.097300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:38.284258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:38.284221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:38.385345 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:38.385308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:38.990848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:38.990809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:39.296818 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:39.296736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:14:39.297082 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:39.297062    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:14:39.314781 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:39.314729    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d" podStartSLOduration=5.314702717 podStartE2EDuration="5.314702717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:35.361956181 +0000 UTC m=+762.773973077" watchObservedRunningTime="2026-09-16 12:14:39.314702717 +0000 UTC m=+766.726719616"
Sep 16 12:14:43.380624 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.380560    2577 generic.go:356] "Generic (PLEG): container finished" podID="619093be-c45c-48cd-a541-f6eb6df4323c" containerID="6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4" exitCode=0
Sep 16 12:14:43.381009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.380633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" event={"ID":"619093be-c45c-48cd-a541-f6eb6df4323c","Type":"ContainerDied","Data":"6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4"}
Sep 16 12:14:43.381053 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.381012    2577 scope.go:117] "RemoveContainer" containerID="6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4"
Sep 16 12:14:43.382276 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.382243    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:14:43.382372 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.382330    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:14:43.382423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.382378    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:14:43.823948 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.823849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:43.923463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:43.923430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:44.389287 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:44.389254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" event={"ID":"619093be-c45c-48cd-a541-f6eb6df4323c","Type":"ContainerStarted","Data":"24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86"}
Sep 16 12:14:44.449525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:44.449489    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:44.449775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:44.449633    2577 patch_prober.go:28] interesting pod/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 12:14:44.449775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:44.449694    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 12:14:44.691377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:44.691296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:14:45.449072 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:45.449032    2577 patch_prober.go:28] interesting pod/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 12:14:45.449674 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:45.449639    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 12:14:46.336796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:46.336763    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:14:46.453029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:46.452979    2577 patch_prober.go:28] interesting pod/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 12:14:46.453527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:46.453050    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 12:14:47.343198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:47.343167    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:14:47.450647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:47.449995    2577 patch_prober.go:28] interesting pod/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 12:14:47.450647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:47.450061    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 12:14:47.674768 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:47.674733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:48.449752 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:48.449712    2577 patch_prober.go:28] interesting pod/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 12:14:48.450054 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:48.450024    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 12:14:48.973451 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:48.973418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:49.449974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:49.449931    2577 patch_prober.go:28] interesting pod/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 12:14:49.450272 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:49.450242    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 12:14:49.819723 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:49.819625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:50.245707 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:50.245650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:50.453488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:50.453454    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:50.454422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:50.454401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:14:50.496110 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:50.496024    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:51.375664 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:51.375592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:52.281236 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:52.281203    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:52.841814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:52.841785    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"]
Sep 16 12:14:52.844122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:52.844106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"
Sep 16 12:14:52.846867 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:52.846840    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:14:52.847251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:52.846955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:14:52.854008 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:52.853832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"]
Sep 16 12:14:52.920322 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:52.920279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e771810-3126-45ce-b2b9-ea1ffa28ff9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67e03001-grt7v\" (UID: \"6e771810-3126-45ce-b2b9-ea1ffa28ff9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"
Sep 16 12:14:53.020986 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.020941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e771810-3126-45ce-b2b9-ea1ffa28ff9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67e03001-grt7v\" (UID: \"6e771810-3126-45ce-b2b9-ea1ffa28ff9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"
Sep 16 12:14:53.023348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.023325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:14:53.031862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.031826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e771810-3126-45ce-b2b9-ea1ffa28ff9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67e03001-grt7v\" (UID: \"6e771810-3126-45ce-b2b9-ea1ffa28ff9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"
Sep 16 12:14:53.076805 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.076768    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:53.164740 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.164708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:14:53.173146 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.173124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"
Sep 16 12:14:53.444808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.444776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"]
Sep 16 12:14:53.445890 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:14:53.445856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e771810_3126_45ce_b2b9_ea1ffa28ff9a.slice/crio-f4101658f4c0cdd620227360887eec0792bccffca803c38ae365d4efd01430d8 WatchSource:0}: Error finding container f4101658f4c0cdd620227360887eec0792bccffca803c38ae365d4efd01430d8: Status 404 returned error can't find the container with id f4101658f4c0cdd620227360887eec0792bccffca803c38ae365d4efd01430d8
Sep 16 12:14:53.793778 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:53.793694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:54.358101 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:54.358064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:54.443622 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:54.443583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v" event={"ID":"6e771810-3126-45ce-b2b9-ea1ffa28ff9a","Type":"ContainerStarted","Data":"f4101658f4c0cdd620227360887eec0792bccffca803c38ae365d4efd01430d8"}
Sep 16 12:14:54.458223 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:54.458187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:55.164494 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:55.164462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:55.973888 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:55.973853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:56.459709 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:56.459657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:56.461770 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:56.461738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v" event={"ID":"6e771810-3126-45ce-b2b9-ea1ffa28ff9a","Type":"ContainerStarted","Data":"2b3816376650950414974bd52256911498e88bc4c69c4b2dcf6c0362351e9ccb"}
Sep 16 12:14:56.477372 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:56.477309    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v" podStartSLOduration=2.262912446 podStartE2EDuration="4.477291601s" totalImagesPullingTime="2.214379155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:56.476816134 +0000 UTC m=+783.888833031" watchObservedRunningTime="2026-09-16 12:14:56.477291601 +0000 UTC m=+783.889308497"
Sep 16 12:14:56.676553 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:56.676516    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:57.468838 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:57.468803    2577 generic.go:356] "Generic (PLEG): container finished" podID="6e771810-3126-45ce-b2b9-ea1ffa28ff9a" containerID="2b3816376650950414974bd52256911498e88bc4c69c4b2dcf6c0362351e9ccb" exitCode=0
Sep 16 12:14:57.469326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:57.468867    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v" event={"ID":"6e771810-3126-45ce-b2b9-ea1ffa28ff9a","Type":"ContainerDied","Data":"2b3816376650950414974bd52256911498e88bc4c69c4b2dcf6c0362351e9ccb"}
Sep 16 12:14:57.479968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:57.479935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:58.178933 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:58.178897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:14:58.596502 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:58.596478    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"
Sep 16 12:14:58.668397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:58.668350    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e771810-3126-45ce-b2b9-ea1ffa28ff9a-maas-api-service-ca\") pod \"6e771810-3126-45ce-b2b9-ea1ffa28ff9a\" (UID: \"6e771810-3126-45ce-b2b9-ea1ffa28ff9a\") "
Sep 16 12:14:58.668721 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:58.668697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e771810-3126-45ce-b2b9-ea1ffa28ff9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e771810-3126-45ce-b2b9-ea1ffa28ff9a" (UID: "6e771810-3126-45ce-b2b9-ea1ffa28ff9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:14:58.769487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:58.769397    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e771810-3126-45ce-b2b9-ea1ffa28ff9a-maas-api-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:14:59.479482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:59.479452    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"
Sep 16 12:14:59.479711 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:59.479451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v" event={"ID":"6e771810-3126-45ce-b2b9-ea1ffa28ff9a","Type":"ContainerDied","Data":"f4101658f4c0cdd620227360887eec0792bccffca803c38ae365d4efd01430d8"}
Sep 16 12:14:59.479711 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:14:59.479598    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4101658f4c0cdd620227360887eec0792bccffca803c38ae365d4efd01430d8"
Sep 16 12:15:00.137786 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.137748    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"]
Sep 16 12:15:00.138310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.138289    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e771810-3126-45ce-b2b9-ea1ffa28ff9a" containerName="revoke-keys"
Sep 16 12:15:00.138395 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.138314    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e771810-3126-45ce-b2b9-ea1ffa28ff9a" containerName="revoke-keys"
Sep 16 12:15:00.138448 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.138426    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e771810-3126-45ce-b2b9-ea1ffa28ff9a" containerName="revoke-keys"
Sep 16 12:15:00.141124 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.141101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"
Sep 16 12:15:00.141874 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.141849    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf"]
Sep 16 12:15:00.143979 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.143950    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"]
Sep 16 12:15:00.144083 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.144025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf"
Sep 16 12:15:00.144194 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.144098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:15:00.146144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.146124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"
Sep 16 12:15:00.148932 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.148868    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"]
Sep 16 12:15:00.164478 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.164448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf"]
Sep 16 12:15:00.164792 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.164770    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"]
Sep 16 12:15:00.165371 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.165347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"
Sep 16 12:15:00.177028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.175611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf"
Sep 16 12:15:00.177028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.176057    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"]
Sep 16 12:15:00.188527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.188497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"
Sep 16 12:15:00.188883 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.188864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"]
Sep 16 12:15:00.188999 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.188979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"
Sep 16 12:15:00.235940 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.235128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"
Sep 16 12:15:00.463859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.463833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"]
Sep 16 12:15:00.464760 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:00.464735    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c13b39_379c_495b_8839_dd5ef9553140.slice/crio-bbc0a4d503a832e4c845c93b31132fb60700de923e120f19010522aa44222d98 WatchSource:0}: Error finding container bbc0a4d503a832e4c845c93b31132fb60700de923e120f19010522aa44222d98: Status 404 returned error can't find the container with id bbc0a4d503a832e4c845c93b31132fb60700de923e120f19010522aa44222d98
Sep 16 12:15:00.468557 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.468533    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf"]
Sep 16 12:15:00.469841 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:00.469819    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbff420_1683_480f_88bc_b5015ba682ec.slice/crio-331f85aa976606e52a1e4bba805895c372a1dae0041732b2d5cc616af2c034cd WatchSource:0}: Error finding container 331f85aa976606e52a1e4bba805895c372a1dae0041732b2d5cc616af2c034cd: Status 404 returned error can't find the container with id 331f85aa976606e52a1e4bba805895c372a1dae0041732b2d5cc616af2c034cd
Sep 16 12:15:00.477192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.477173    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"]
Sep 16 12:15:00.479979 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:00.479954    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca48e89d_622a_4486_bc2e_f76c09fc2cd4.slice/crio-a53f140327c5359d576520d0c3776f174c5a993ba12af8796cac44a1c5c7692f WatchSource:0}: Error finding container a53f140327c5359d576520d0c3776f174c5a993ba12af8796cac44a1c5c7692f: Status 404 returned error can't find the container with id a53f140327c5359d576520d0c3776f174c5a993ba12af8796cac44a1c5c7692f
Sep 16 12:15:00.485118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.485084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf" event={"ID":"3fbff420-1683-480f-88bc-b5015ba682ec","Type":"ContainerStarted","Data":"331f85aa976606e52a1e4bba805895c372a1dae0041732b2d5cc616af2c034cd"}
Sep 16 12:15:00.486488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.486454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt" event={"ID":"92c13b39-379c-495b-8839-dd5ef9553140","Type":"ContainerStarted","Data":"bbc0a4d503a832e4c845c93b31132fb60700de923e120f19010522aa44222d98"}
Sep 16 12:15:00.531686 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:00.531663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"]
Sep 16 12:15:00.533146 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:00.533119    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376911dd_a23c_4283_8aeb_05d30cd34cba.slice/crio-55bfa6e0cb15dc11a4a58cdf809a1679efdbff55e14b75797c5a193006910b13 WatchSource:0}: Error finding container 55bfa6e0cb15dc11a4a58cdf809a1679efdbff55e14b75797c5a193006910b13: Status 404 returned error can't find the container with id 55bfa6e0cb15dc11a4a58cdf809a1679efdbff55e14b75797c5a193006910b13
Sep 16 12:15:01.493356 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.493318    2577 generic.go:356] "Generic (PLEG): container finished" podID="92c13b39-379c-495b-8839-dd5ef9553140" containerID="72486e5e84eb2a9a05098ec51f21784314f3aa9fef21594e0145d299e5529789" exitCode=0
Sep 16 12:15:01.493747 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.493406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt" event={"ID":"92c13b39-379c-495b-8839-dd5ef9553140","Type":"ContainerDied","Data":"72486e5e84eb2a9a05098ec51f21784314f3aa9fef21594e0145d299e5529789"}
Sep 16 12:15:01.494965 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.494939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf" event={"ID":"3fbff420-1683-480f-88bc-b5015ba682ec","Type":"ContainerStarted","Data":"cea6469a022bffca9672b469a5357d41d7245e3a16fddb44f82d4195c6dc5162"}
Sep 16 12:15:01.496444 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.496417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt" event={"ID":"ca48e89d-622a-4486-bc2e-f76c09fc2cd4","Type":"ContainerStarted","Data":"5d49d769c3810e4dfdea1c9cf3f9063b35df8a3e532b4239e77b00dc474b084e"}
Sep 16 12:15:01.496550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.496470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt" event={"ID":"ca48e89d-622a-4486-bc2e-f76c09fc2cd4","Type":"ContainerStarted","Data":"a53f140327c5359d576520d0c3776f174c5a993ba12af8796cac44a1c5c7692f"}
Sep 16 12:15:01.497935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.497913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc" event={"ID":"376911dd-a23c-4283-8aeb-05d30cd34cba","Type":"ContainerStarted","Data":"98b53940084d6c72d11465bdfefd8584097c090ec4ae9efb0600938d5e3aaaa9"}
Sep 16 12:15:01.498023 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.497948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc" event={"ID":"376911dd-a23c-4283-8aeb-05d30cd34cba","Type":"ContainerStarted","Data":"55bfa6e0cb15dc11a4a58cdf809a1679efdbff55e14b75797c5a193006910b13"}
Sep 16 12:15:01.528256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.528206    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc" podStartSLOduration=1.528190696 podStartE2EDuration="1.528190696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:01.527504595 +0000 UTC m=+788.939521492" watchObservedRunningTime="2026-09-16 12:15:01.528190696 +0000 UTC m=+788.940207646"
Sep 16 12:15:01.542118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.542072    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf" podStartSLOduration=1.542053536 podStartE2EDuration="1.542053536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:01.541181446 +0000 UTC m=+788.953198342" watchObservedRunningTime="2026-09-16 12:15:01.542053536 +0000 UTC m=+788.954070431"
Sep 16 12:15:01.555748 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:01.555700    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt" podStartSLOduration=1.555685565 podStartE2EDuration="1.555685565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:01.553763231 +0000 UTC m=+788.965780139" watchObservedRunningTime="2026-09-16 12:15:01.555685565 +0000 UTC m=+788.967702460"
Sep 16 12:15:02.504101 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:02.504066    2577 generic.go:356] "Generic (PLEG): container finished" podID="3fbff420-1683-480f-88bc-b5015ba682ec" containerID="cea6469a022bffca9672b469a5357d41d7245e3a16fddb44f82d4195c6dc5162" exitCode=0
Sep 16 12:15:02.504532 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:02.504161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf" event={"ID":"3fbff420-1683-480f-88bc-b5015ba682ec","Type":"ContainerDied","Data":"cea6469a022bffca9672b469a5357d41d7245e3a16fddb44f82d4195c6dc5162"}
Sep 16 12:15:02.505695 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:02.505669    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca48e89d-622a-4486-bc2e-f76c09fc2cd4" containerID="5d49d769c3810e4dfdea1c9cf3f9063b35df8a3e532b4239e77b00dc474b084e" exitCode=0
Sep 16 12:15:02.505818 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:02.505763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt" event={"ID":"ca48e89d-622a-4486-bc2e-f76c09fc2cd4","Type":"ContainerDied","Data":"5d49d769c3810e4dfdea1c9cf3f9063b35df8a3e532b4239e77b00dc474b084e"}
Sep 16 12:15:02.507140 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:02.507121    2577 generic.go:356] "Generic (PLEG): container finished" podID="376911dd-a23c-4283-8aeb-05d30cd34cba" containerID="98b53940084d6c72d11465bdfefd8584097c090ec4ae9efb0600938d5e3aaaa9" exitCode=0
Sep 16 12:15:02.507232 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:02.507149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc" event={"ID":"376911dd-a23c-4283-8aeb-05d30cd34cba","Type":"ContainerDied","Data":"98b53940084d6c72d11465bdfefd8584097c090ec4ae9efb0600938d5e3aaaa9"}
Sep 16 12:15:02.650802 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:02.650774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"
Sep 16 12:15:03.512841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:03.512813    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"
Sep 16 12:15:03.513318 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:03.512856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt" event={"ID":"92c13b39-379c-495b-8839-dd5ef9553140","Type":"ContainerDied","Data":"bbc0a4d503a832e4c845c93b31132fb60700de923e120f19010522aa44222d98"}
Sep 16 12:15:03.513318 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:03.512905    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbc0a4d503a832e4c845c93b31132fb60700de923e120f19010522aa44222d98"
Sep 16 12:15:03.648090 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:03.648061    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf"
Sep 16 12:15:03.686978 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:03.686935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:03.703056 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:03.703031    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"
Sep 16 12:15:03.705824 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:03.705802    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"
Sep 16 12:15:04.250639 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.250610    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"]
Sep 16 12:15:04.252694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.252666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:04.259054 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.259024    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"]
Sep 16 12:15:04.278235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.278202    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"]
Sep 16 12:15:04.278469 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.278442    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm" podUID="cbef05b4-7b67-4b44-8175-ae1c05e48fd2" containerName="payload-processing" containerID="cri-o://4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924" gracePeriod=30
Sep 16 12:15:04.284187 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.284160    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"]
Sep 16 12:15:04.284396 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.284373    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p" podUID="8580422c-9106-42e1-ad92-a7a9fef6c64c" containerName="payload-pre-processing" containerID="cri-o://7e4a57e88997f09b43c5e8c488687cd332a84ec67c5cb0efda4641bc97687039" gracePeriod=30
Sep 16 12:15:04.320121 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.320076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:04.382883 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.382855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:04.422529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.422494    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"]
Sep 16 12:15:04.424214 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.424189    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67e03001-grt7v"]
Sep 16 12:15:04.521641 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.521543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:04.522842 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.522788    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:15:04.523038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.522892    2577 generic.go:356] "Generic (PLEG): container finished" podID="cbef05b4-7b67-4b44-8175-ae1c05e48fd2" containerID="4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924" exitCode=0
Sep 16 12:15:04.523038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.522996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm" event={"ID":"cbef05b4-7b67-4b44-8175-ae1c05e48fd2","Type":"ContainerDied","Data":"4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924"}
Sep 16 12:15:04.523205 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.523046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm" event={"ID":"cbef05b4-7b67-4b44-8175-ae1c05e48fd2","Type":"ContainerDied","Data":"bb87f7f32679222289963e38530952d23faa71f37c8a085c79ef98f9355a1aca"}
Sep 16 12:15:04.523205 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.523122    2577 scope.go:117] "RemoveContainer" containerID="4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924"
Sep 16 12:15:04.525504 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.525478    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"
Sep 16 12:15:04.534814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.525519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc" event={"ID":"376911dd-a23c-4283-8aeb-05d30cd34cba","Type":"ContainerDied","Data":"55bfa6e0cb15dc11a4a58cdf809a1679efdbff55e14b75797c5a193006910b13"}
Sep 16 12:15:04.534814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.525600    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bfa6e0cb15dc11a4a58cdf809a1679efdbff55e14b75797c5a193006910b13"
Sep 16 12:15:04.534814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.528278    2577 generic.go:356] "Generic (PLEG): container finished" podID="8580422c-9106-42e1-ad92-a7a9fef6c64c" containerID="7e4a57e88997f09b43c5e8c488687cd332a84ec67c5cb0efda4641bc97687039" exitCode=0
Sep 16 12:15:04.534814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.528349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p" event={"ID":"8580422c-9106-42e1-ad92-a7a9fef6c64c","Type":"ContainerDied","Data":"7e4a57e88997f09b43c5e8c488687cd332a84ec67c5cb0efda4641bc97687039"}
Sep 16 12:15:04.539583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.536688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf" event={"ID":"3fbff420-1683-480f-88bc-b5015ba682ec","Type":"ContainerDied","Data":"331f85aa976606e52a1e4bba805895c372a1dae0041732b2d5cc616af2c034cd"}
Sep 16 12:15:04.539583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.536740    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="331f85aa976606e52a1e4bba805895c372a1dae0041732b2d5cc616af2c034cd"
Sep 16 12:15:04.539583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.536812    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826015-hmmpf"
Sep 16 12:15:04.548682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.548123    2577 scope.go:117] "RemoveContainer" containerID="4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924"
Sep 16 12:15:04.549376 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:15:04.548963    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924\": container with ID starting with 4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924 not found: ID does not exist" containerID="4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924"
Sep 16 12:15:04.549376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.549011    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924"} err="failed to get container status \"4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924\": rpc error: code = NotFound desc = could not find container \"4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924\": container with ID starting with 4e800b13ee4b0212b6770b70a1eeee9ae681fc9d37ce9468bc8940f323d39924 not found: ID does not exist"
Sep 16 12:15:04.549376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.549077    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53f140327c5359d576520d0c3776f174c5a993ba12af8796cac44a1c5c7692f"
Sep 16 12:15:04.549376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.549136    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"
Sep 16 12:15:04.573350 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:15:04.573192    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="ca48e89d-622a-4486-bc2e-f76c09fc2cd4" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"
Sep 16 12:15:04.582076 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.582047    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:15:04.584562 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:15:04.584526    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt\" is forbidden: User \"system:node:ip-10-0-133-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-38.ec2.internal' and this object" podUID="ca48e89d-622a-4486-bc2e-f76c09fc2cd4" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-67e03001-29826015-c58qt"
Sep 16 12:15:04.620751 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.620720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9jh2\" (UniqueName: \"kubernetes.io/projected/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-kube-api-access-r9jh2\") pod \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") "
Sep 16 12:15:04.620928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.620765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8580422c-9106-42e1-ad92-a7a9fef6c64c-plugins-config-volume\") pod \"8580422c-9106-42e1-ad92-a7a9fef6c64c\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") "
Sep 16 12:15:04.620928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.620792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx9h4\" (UniqueName: \"kubernetes.io/projected/8580422c-9106-42e1-ad92-a7a9fef6c64c-kube-api-access-vx9h4\") pod \"8580422c-9106-42e1-ad92-a7a9fef6c64c\" (UID: \"8580422c-9106-42e1-ad92-a7a9fef6c64c\") "
Sep 16 12:15:04.620928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.620813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-plugins-config-volume\") pod \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\" (UID: \"cbef05b4-7b67-4b44-8175-ae1c05e48fd2\") "
Sep 16 12:15:04.621216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.621187    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8580422c-9106-42e1-ad92-a7a9fef6c64c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8580422c-9106-42e1-ad92-a7a9fef6c64c" (UID: "8580422c-9106-42e1-ad92-a7a9fef6c64c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:04.621278 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.621203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbef05b4-7b67-4b44-8175-ae1c05e48fd2" (UID: "cbef05b4-7b67-4b44-8175-ae1c05e48fd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:04.623005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.622976    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-kube-api-access-r9jh2" (OuterVolumeSpecName: "kube-api-access-r9jh2") pod "cbef05b4-7b67-4b44-8175-ae1c05e48fd2" (UID: "cbef05b4-7b67-4b44-8175-ae1c05e48fd2"). InnerVolumeSpecName "kube-api-access-r9jh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:04.623282 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.623035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8580422c-9106-42e1-ad92-a7a9fef6c64c-kube-api-access-vx9h4" (OuterVolumeSpecName: "kube-api-access-vx9h4") pod "8580422c-9106-42e1-ad92-a7a9fef6c64c" (UID: "8580422c-9106-42e1-ad92-a7a9fef6c64c"). InnerVolumeSpecName "kube-api-access-vx9h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:04.722550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.722511    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.722550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.722541    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9jh2\" (UniqueName: \"kubernetes.io/projected/cbef05b4-7b67-4b44-8175-ae1c05e48fd2-kube-api-access-r9jh2\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.722550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.722550    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8580422c-9106-42e1-ad92-a7a9fef6c64c-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.722550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.722558    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx9h4\" (UniqueName: \"kubernetes.io/projected/8580422c-9106-42e1-ad92-a7a9fef6c64c-kube-api-access-vx9h4\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:04.789895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.789811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:04.890995 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:04.890949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:05.112088 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.112004    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e771810-3126-45ce-b2b9-ea1ffa28ff9a" path="/var/lib/kubelet/pods/6e771810-3126-45ce-b2b9-ea1ffa28ff9a/volumes"
Sep 16 12:15:05.112305 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.112292    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca48e89d-622a-4486-bc2e-f76c09fc2cd4" path="/var/lib/kubelet/pods/ca48e89d-622a-4486-bc2e-f76c09fc2cd4/volumes"
Sep 16 12:15:05.296619 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.296549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:05.557339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.557303    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"
Sep 16 12:15:05.557339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.557326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p" event={"ID":"8580422c-9106-42e1-ad92-a7a9fef6c64c","Type":"ContainerDied","Data":"35640131f7bfe5bec1101f0de9371773e629f2f6aa9bf39dfb004b63bd5b2639"}
Sep 16 12:15:05.557888 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.557395    2577 scope.go:117] "RemoveContainer" containerID="7e4a57e88997f09b43c5e8c488687cd332a84ec67c5cb0efda4641bc97687039"
Sep 16 12:15:05.558964 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.558704    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"
Sep 16 12:15:05.573596 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.573547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:05.613446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.613371    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"]
Sep 16 12:15:05.616902 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.616870    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67e03001-6c4bd769d7-tkzzm"]
Sep 16 12:15:05.626392 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.626363    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"]
Sep 16 12:15:05.626995 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.626962    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="376911dd-a23c-4283-8aeb-05d30cd34cba" containerName="cleanup"
Sep 16 12:15:05.626995 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.626985    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="376911dd-a23c-4283-8aeb-05d30cd34cba" containerName="cleanup"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627040    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbef05b4-7b67-4b44-8175-ae1c05e48fd2" containerName="payload-processing"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627049    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbef05b4-7b67-4b44-8175-ae1c05e48fd2" containerName="payload-processing"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c13b39-379c-495b-8839-dd5ef9553140" containerName="cleanup"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627094    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c13b39-379c-495b-8839-dd5ef9553140" containerName="cleanup"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627106    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8580422c-9106-42e1-ad92-a7a9fef6c64c" containerName="payload-pre-processing"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627114    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8580422c-9106-42e1-ad92-a7a9fef6c64c" containerName="payload-pre-processing"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627138    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca48e89d-622a-4486-bc2e-f76c09fc2cd4" containerName="cleanup"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627146    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca48e89d-622a-4486-bc2e-f76c09fc2cd4" containerName="cleanup"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbff420-1683-480f-88bc-b5015ba682ec" containerName="cleanup"
Sep 16 12:15:05.627242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627167    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbff420-1683-480f-88bc-b5015ba682ec" containerName="cleanup"
Sep 16 12:15:05.627734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627249    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c13b39-379c-495b-8839-dd5ef9553140" containerName="cleanup"
Sep 16 12:15:05.627734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627261    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbef05b4-7b67-4b44-8175-ae1c05e48fd2" containerName="payload-processing"
Sep 16 12:15:05.627734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627271    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbff420-1683-480f-88bc-b5015ba682ec" containerName="cleanup"
Sep 16 12:15:05.627734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627282    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca48e89d-622a-4486-bc2e-f76c09fc2cd4" containerName="cleanup"
Sep 16 12:15:05.627734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627290    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="376911dd-a23c-4283-8aeb-05d30cd34cba" containerName="cleanup"
Sep 16 12:15:05.627734 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.627299    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8580422c-9106-42e1-ad92-a7a9fef6c64c" containerName="payload-pre-processing"
Sep 16 12:15:05.631648 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.631617    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.637634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.637611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f74dc7e1-gw-openshift-default-dockercfg-jlhrp\""
Sep 16 12:15:05.639061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.639036    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"]
Sep 16 12:15:05.646782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.646664    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67e03001-778cfd66b6-cq76p"]
Sep 16 12:15:05.648747 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.648725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"]
Sep 16 12:15:05.667870 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.667842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-data\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-credential-socket\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-certs\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-podinfo\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-token\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istiod-ca-cert\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-socket\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdjhl\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-kube-api-access-sdjhl\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.731595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.731379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-envoy\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.799900 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.799861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:05.832348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-socket\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdjhl\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-kube-api-access-sdjhl\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-envoy\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-data\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-credential-socket\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-certs\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-podinfo\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-token\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.832649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istiod-ca-cert\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.833022 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.832727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-socket\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.833309 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.833281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istiod-ca-cert\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.833437 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.833374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-data\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.833437 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.833423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-certs\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.833561 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.833485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-credential-socket\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.834764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.834742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-envoy\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.835286 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.835265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-podinfo\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.839979 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.839948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-token\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.840773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.840749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdjhl\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-kube-api-access-sdjhl\") pod \"e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") " pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:05.903996 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.903959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:05.944353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:05.944316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:06.218869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.218844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"]
Sep 16 12:15:06.220401 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:06.220373    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7471ed9a_f5fe_4646_812e_89db3f29a2c0.slice/crio-bba09a5d3292d38fa59d712bbab9bcb6bcdebd3448a4d2d7505360d3859085ed WatchSource:0}: Error finding container bba09a5d3292d38fa59d712bbab9bcb6bcdebd3448a4d2d7505360d3859085ed: Status 404 returned error can't find the container with id bba09a5d3292d38fa59d712bbab9bcb6bcdebd3448a4d2d7505360d3859085ed
Sep 16 12:15:06.222590 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.222541    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:06.222682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.222620    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:06.222682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.222649    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:06.480010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.479922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:06.567500 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.567457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" event={"ID":"7471ed9a-f5fe-4646-812e-89db3f29a2c0","Type":"ContainerStarted","Data":"73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a"}
Sep 16 12:15:06.567991 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.567532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" event={"ID":"7471ed9a-f5fe-4646-812e-89db3f29a2c0","Type":"ContainerStarted","Data":"bba09a5d3292d38fa59d712bbab9bcb6bcdebd3448a4d2d7505360d3859085ed"}
Sep 16 12:15:06.587517 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.587471    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" podStartSLOduration=1.587456035 podStartE2EDuration="1.587456035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:06.586380488 +0000 UTC m=+793.998397385" watchObservedRunningTime="2026-09-16 12:15:06.587456035 +0000 UTC m=+793.999472931"
Sep 16 12:15:06.945459 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:06.945417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:07.111558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:07.111519    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8580422c-9106-42e1-ad92-a7a9fef6c64c" path="/var/lib/kubelet/pods/8580422c-9106-42e1-ad92-a7a9fef6c64c/volumes"
Sep 16 12:15:07.111932 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:07.111916    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbef05b4-7b67-4b44-8175-ae1c05e48fd2" path="/var/lib/kubelet/pods/cbef05b4-7b67-4b44-8175-ae1c05e48fd2/volumes"
Sep 16 12:15:07.206067 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:07.205978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:07.873594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:07.873535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:07.950532 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:07.950502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:07.982829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:07.982801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:08.082114 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:08.082083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:08.182116 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:08.182086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:08.578560 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:08.578477    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:08.579372 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:08.579347    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:08.791841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:08.791809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:09.186579 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:09.186542    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:15:09.426547 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:09.426517    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:09.525345 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:09.525269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:09.872394 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:09.872294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:09.967893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:09.967862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:09.999779 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:09.999748    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"]
Sep 16 12:15:10.004697 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.004674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.007525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.007502    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f74dc7e1-dockercfg-fldkm\""
Sep 16 12:15:10.008044 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.008027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f74dc7e1\""
Sep 16 12:15:10.019867 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.019840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"]
Sep 16 12:15:10.074444 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.074401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zjp\" (UniqueName: \"kubernetes.io/projected/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-kube-api-access-s8zjp\") pod \"payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") " pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.074652 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.074469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") " pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.076520 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.076493    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"]
Sep 16 12:15:10.079850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.079830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.104939 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.104491    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"]
Sep 16 12:15:10.175834 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.175790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5116c5b0-260e-4333-818b-ec2553c370b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.176027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.175870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjsqd\" (UniqueName: \"kubernetes.io/projected/5116c5b0-260e-4333-818b-ec2553c370b5-kube-api-access-xjsqd\") pod \"payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.176027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.175891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zjp\" (UniqueName: \"kubernetes.io/projected/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-kube-api-access-s8zjp\") pod \"payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") " pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.176027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.175919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") " pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.176628 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.176605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") " pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.184011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.183981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zjp\" (UniqueName: \"kubernetes.io/projected/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-kube-api-access-s8zjp\") pod \"payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") " pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.276660 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.276622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5116c5b0-260e-4333-818b-ec2553c370b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.277016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.276693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjsqd\" (UniqueName: \"kubernetes.io/projected/5116c5b0-260e-4333-818b-ec2553c370b5-kube-api-access-xjsqd\") pod \"payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.277405 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.277386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5116c5b0-260e-4333-818b-ec2553c370b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.284646 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.284619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjsqd\" (UniqueName: \"kubernetes.io/projected/5116c5b0-260e-4333-818b-ec2553c370b5-kube-api-access-xjsqd\") pod \"payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.315219 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.315183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:10.378947 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.378412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:10.390903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.390875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:10.593017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.592994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"]
Sep 16 12:15:10.595001 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:10.594965    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6b5a5_96e8_4a93_9abd_c8e0b11e931a.slice/crio-d90499a60d7698b364b6af09b059986e95278d01b70664dad12d9972db00548a WatchSource:0}: Error finding container d90499a60d7698b364b6af09b059986e95278d01b70664dad12d9972db00548a: Status 404 returned error can't find the container with id d90499a60d7698b364b6af09b059986e95278d01b70664dad12d9972db00548a
Sep 16 12:15:10.676404 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.676381    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"]
Sep 16 12:15:10.678849 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:10.678819    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5116c5b0_260e_4333_818b_ec2553c370b5.slice/crio-9947182c950ef41e800a07a2f25febb84add9fa74a8843ec5b09622e7f301269 WatchSource:0}: Error finding container 9947182c950ef41e800a07a2f25febb84add9fa74a8843ec5b09622e7f301269: Status 404 returned error can't find the container with id 9947182c950ef41e800a07a2f25febb84add9fa74a8843ec5b09622e7f301269
Sep 16 12:15:10.757124 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:10.757087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:11.078741 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.078645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:11.596551 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.596511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk" event={"ID":"5116c5b0-260e-4333-818b-ec2553c370b5","Type":"ContainerStarted","Data":"af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8"}
Sep 16 12:15:11.597012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.596706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:11.597012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.596743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk" event={"ID":"5116c5b0-260e-4333-818b-ec2553c370b5","Type":"ContainerStarted","Data":"9947182c950ef41e800a07a2f25febb84add9fa74a8843ec5b09622e7f301269"}
Sep 16 12:15:11.598109 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.598079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw" event={"ID":"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a","Type":"ContainerStarted","Data":"f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b"}
Sep 16 12:15:11.598237 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.598121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw" event={"ID":"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a","Type":"ContainerStarted","Data":"d90499a60d7698b364b6af09b059986e95278d01b70664dad12d9972db00548a"}
Sep 16 12:15:11.598237 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.598142    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:11.614611 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.614555    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk" podStartSLOduration=1.614540843 podStartE2EDuration="1.614540843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:11.612678631 +0000 UTC m=+799.024695529" watchObservedRunningTime="2026-09-16 12:15:11.614540843 +0000 UTC m=+799.026557780"
Sep 16 12:15:11.630502 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.630446    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw" podStartSLOduration=2.630426856 podStartE2EDuration="2.630426856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:11.628657103 +0000 UTC m=+799.040674002" watchObservedRunningTime="2026-09-16 12:15:11.630426856 +0000 UTC m=+799.042443753"
Sep 16 12:15:11.788149 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:11.788105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:12.485044 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:12.484993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:22.606178 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:22.606081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:22.606596 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:22.606184    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:26.064463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:26.064427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:26.286061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:26.286019    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:15:28.700895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:28.700862    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerID="73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a" exitCode=0
Sep 16 12:15:28.701322 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:28.700937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" event={"ID":"7471ed9a-f5fe-4646-812e-89db3f29a2c0","Type":"ContainerDied","Data":"73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a"}
Sep 16 12:15:28.701381 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:28.701317    2577 scope.go:117] "RemoveContainer" containerID="73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a"
Sep 16 12:15:28.708375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:28.708323    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:28.708446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:28.708430    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:28.708483 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:28.708468    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:29.708317 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:29.708284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" event={"ID":"7471ed9a-f5fe-4646-812e-89db3f29a2c0","Type":"ContainerStarted","Data":"8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342"}
Sep 16 12:15:29.945385 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:29.945349    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:29.945782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:29.945691    2577 patch_prober.go:28] interesting pod/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 16 12:15:29.945949 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:29.945927    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 16 12:15:30.945680 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:30.945632    2577 patch_prober.go:28] interesting pod/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 16 12:15:30.946177 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:30.945711    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 16 12:15:31.269970 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:31.269878    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:31.928214 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:31.928181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:31.945769 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:31.945729    2577 patch_prober.go:28] interesting pod/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 16 12:15:31.946244 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:31.945797    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 16 12:15:32.547256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:32.547223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:32.945736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:32.945694    2577 patch_prober.go:28] interesting pod/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 16 12:15:32.946045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:32.946016    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 16 12:15:33.034926 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.034872    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"]
Sep 16 12:15:33.040418 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.040390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"
Sep 16 12:15:33.043541 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.043515    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:15:33.043971 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.043953    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:15:33.048919 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.048845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"]
Sep 16 12:15:33.186590 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.186526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36da08cf-d986-4a4a-bb81-8b7a83c32ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84\" (UID: \"36da08cf-d986-4a4a-bb81-8b7a83c32ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"
Sep 16 12:15:33.288104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.288000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36da08cf-d986-4a4a-bb81-8b7a83c32ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84\" (UID: \"36da08cf-d986-4a4a-bb81-8b7a83c32ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"
Sep 16 12:15:33.288689 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.288668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36da08cf-d986-4a4a-bb81-8b7a83c32ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84\" (UID: \"36da08cf-d986-4a4a-bb81-8b7a83c32ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"
Sep 16 12:15:33.359246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.359211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"
Sep 16 12:15:33.370796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.370762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:33.631334 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.631297    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"]
Sep 16 12:15:33.739419 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.739380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84" event={"ID":"36da08cf-d986-4a4a-bb81-8b7a83c32ea5","Type":"ContainerStarted","Data":"7812c4bd80ef5d7b3dc25067da36187d30e0bbf203875daa4647ee2809cb34b2"}
Sep 16 12:15:33.945350 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.945320    2577 patch_prober.go:28] interesting pod/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 16 12:15:33.945469 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.945375    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 16 12:15:33.980741 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:33.980710    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:34.671073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:34.671034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:34.749263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:34.749221    2577 generic.go:356] "Generic (PLEG): container finished" podID="36da08cf-d986-4a4a-bb81-8b7a83c32ea5" containerID="7b8fe5463fac7ab42d07919b6c6d68714de9aad9677cf25867a6ee1eea96dfa8" exitCode=0
Sep 16 12:15:34.749462 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:34.749332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84" event={"ID":"36da08cf-d986-4a4a-bb81-8b7a83c32ea5","Type":"ContainerDied","Data":"7b8fe5463fac7ab42d07919b6c6d68714de9aad9677cf25867a6ee1eea96dfa8"}
Sep 16 12:15:35.036350 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:35.036252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:35.036801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:35.036776    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:35.039974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:35.039944    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:35.403205 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:35.403173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:35.887291 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:35.887266    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"
Sep 16 12:15:36.013923 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.013828    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36da08cf-d986-4a4a-bb81-8b7a83c32ea5-maas-api-service-ca\") pod \"36da08cf-d986-4a4a-bb81-8b7a83c32ea5\" (UID: \"36da08cf-d986-4a4a-bb81-8b7a83c32ea5\") "
Sep 16 12:15:36.014204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.014180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36da08cf-d986-4a4a-bb81-8b7a83c32ea5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36da08cf-d986-4a4a-bb81-8b7a83c32ea5" (UID: "36da08cf-d986-4a4a-bb81-8b7a83c32ea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:36.113016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.112984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:36.114463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.114443    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36da08cf-d986-4a4a-bb81-8b7a83c32ea5-maas-api-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:36.760461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.760422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84" event={"ID":"36da08cf-d986-4a4a-bb81-8b7a83c32ea5","Type":"ContainerDied","Data":"7812c4bd80ef5d7b3dc25067da36187d30e0bbf203875daa4647ee2809cb34b2"}
Sep 16 12:15:36.760685 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.760505    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7812c4bd80ef5d7b3dc25067da36187d30e0bbf203875daa4647ee2809cb34b2"
Sep 16 12:15:36.760685 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.760431    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"
Sep 16 12:15:36.954279 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:36.954248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:37.541195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:37.541146    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:38.128123 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:38.128046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:39.287193 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.287156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:39.802765 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.802732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:39.834004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.833971    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"]
Sep 16 12:15:39.834449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.834419    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw" podUID="a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" containerName="payload-processing" containerID="cri-o://f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b" gracePeriod=30
Sep 16 12:15:39.839485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.839450    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"]
Sep 16 12:15:39.840049 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.840010    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk" podUID="5116c5b0-260e-4333-818b-ec2553c370b5" containerName="payload-pre-processing" containerID="cri-o://af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8" gracePeriod=30
Sep 16 12:15:39.876954 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.876919    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:39.938890 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.938846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:39.972605 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.972548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"]
Sep 16 12:15:39.974678 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.974651    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f74dc7e1-vhj84"]
Sep 16 12:15:39.998168 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:39.998136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:40.109643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.109164    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"]
Sep 16 12:15:40.109643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.109469    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy" containerID="cri-o://8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342" gracePeriod=30
Sep 16 12:15:40.127820 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.126179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:40.151698 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.151672    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:40.155145 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.155123    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:40.254170 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.254141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8zjp\" (UniqueName: \"kubernetes.io/projected/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-kube-api-access-s8zjp\") pod \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") "
Sep 16 12:15:40.254358 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.254199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5116c5b0-260e-4333-818b-ec2553c370b5-plugins-config-volume\") pod \"5116c5b0-260e-4333-818b-ec2553c370b5\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") "
Sep 16 12:15:40.254358 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.254257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-plugins-config-volume\") pod \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\" (UID: \"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a\") "
Sep 16 12:15:40.254358 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.254287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjsqd\" (UniqueName: \"kubernetes.io/projected/5116c5b0-260e-4333-818b-ec2553c370b5-kube-api-access-xjsqd\") pod \"5116c5b0-260e-4333-818b-ec2553c370b5\" (UID: \"5116c5b0-260e-4333-818b-ec2553c370b5\") "
Sep 16 12:15:40.254693 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.254662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5116c5b0-260e-4333-818b-ec2553c370b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5116c5b0-260e-4333-818b-ec2553c370b5" (UID: "5116c5b0-260e-4333-818b-ec2553c370b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:40.254774 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.254691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" (UID: "a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:40.256357 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.256337    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5116c5b0-260e-4333-818b-ec2553c370b5-kube-api-access-xjsqd" (OuterVolumeSpecName: "kube-api-access-xjsqd") pod "5116c5b0-260e-4333-818b-ec2553c370b5" (UID: "5116c5b0-260e-4333-818b-ec2553c370b5"). InnerVolumeSpecName "kube-api-access-xjsqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:40.256422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.256366    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-kube-api-access-s8zjp" (OuterVolumeSpecName: "kube-api-access-s8zjp") pod "a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" (UID: "a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a"). InnerVolumeSpecName "kube-api-access-s8zjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:40.355308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.355268    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.355308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.355301    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjsqd\" (UniqueName: \"kubernetes.io/projected/5116c5b0-260e-4333-818b-ec2553c370b5-kube-api-access-xjsqd\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.355308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.355314    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zjp\" (UniqueName: \"kubernetes.io/projected/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a-kube-api-access-s8zjp\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.355790 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.355325    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5116c5b0-260e-4333-818b-ec2553c370b5-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:40.393243 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.393192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:40.566426 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.566388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:40.747117 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.747031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:40.790492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.790449    2577 generic.go:356] "Generic (PLEG): container finished" podID="5116c5b0-260e-4333-818b-ec2553c370b5" containerID="af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8" exitCode=0
Sep 16 12:15:40.790702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.790504    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"
Sep 16 12:15:40.790702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.790533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk" event={"ID":"5116c5b0-260e-4333-818b-ec2553c370b5","Type":"ContainerDied","Data":"af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8"}
Sep 16 12:15:40.790702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.790635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk" event={"ID":"5116c5b0-260e-4333-818b-ec2553c370b5","Type":"ContainerDied","Data":"9947182c950ef41e800a07a2f25febb84add9fa74a8843ec5b09622e7f301269"}
Sep 16 12:15:40.790702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.790671    2577 scope.go:117] "RemoveContainer" containerID="af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8"
Sep 16 12:15:40.792418 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.792392    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" containerID="f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b" exitCode=0
Sep 16 12:15:40.792520 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.792443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw" event={"ID":"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a","Type":"ContainerDied","Data":"f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b"}
Sep 16 12:15:40.792520 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.792451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"
Sep 16 12:15:40.792520 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.792481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw" event={"ID":"a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a","Type":"ContainerDied","Data":"d90499a60d7698b364b6af09b059986e95278d01b70664dad12d9972db00548a"}
Sep 16 12:15:40.801548 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.801523    2577 scope.go:117] "RemoveContainer" containerID="af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8"
Sep 16 12:15:40.801884 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:15:40.801862    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8\": container with ID starting with af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8 not found: ID does not exist" containerID="af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8"
Sep 16 12:15:40.801970 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.801895    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8"} err="failed to get container status \"af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8\": rpc error: code = NotFound desc = could not find container \"af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8\": container with ID starting with af4c38a58749cd3c2f5d6fd2de207f1e1f38c2441f70886faea53a5ccb9112f8 not found: ID does not exist"
Sep 16 12:15:40.801970 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.801918    2577 scope.go:117] "RemoveContainer" containerID="f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b"
Sep 16 12:15:40.811233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.811211    2577 scope.go:117] "RemoveContainer" containerID="f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b"
Sep 16 12:15:40.811472 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:15:40.811450    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b\": container with ID starting with f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b not found: ID does not exist" containerID="f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b"
Sep 16 12:15:40.811539 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.811484    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b"} err="failed to get container status \"f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b\": rpc error: code = NotFound desc = could not find container \"f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b\": container with ID starting with f1bb6db1e073306c707f21cb0ff22184c6192c7bd758b083b92d5307abede47b not found: ID does not exist"
Sep 16 12:15:40.820191 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.820166    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"]
Sep 16 12:15:40.827816 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.827787    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f74dc7e1-68b484d56b-vvztk"]
Sep 16 12:15:40.847969 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.847936    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"]
Sep 16 12:15:40.857493 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.857466    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f74dc7e1-64f958bb94-sp9pw"]
Sep 16 12:15:40.873377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.873340    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"]
Sep 16 12:15:40.873951 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.873929    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5116c5b0-260e-4333-818b-ec2553c370b5" containerName="payload-pre-processing"
Sep 16 12:15:40.874011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.873955    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5116c5b0-260e-4333-818b-ec2553c370b5" containerName="payload-pre-processing"
Sep 16 12:15:40.874011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.873968    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36da08cf-d986-4a4a-bb81-8b7a83c32ea5" containerName="revoke-keys"
Sep 16 12:15:40.874011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.873976    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36da08cf-d986-4a4a-bb81-8b7a83c32ea5" containerName="revoke-keys"
Sep 16 12:15:40.874011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.873994    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" containerName="payload-processing"
Sep 16 12:15:40.874011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.874003    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" containerName="payload-processing"
Sep 16 12:15:40.874192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.874092    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" containerName="payload-processing"
Sep 16 12:15:40.874192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.874104    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5116c5b0-260e-4333-818b-ec2553c370b5" containerName="payload-pre-processing"
Sep 16 12:15:40.874192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.874114    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36da08cf-d986-4a4a-bb81-8b7a83c32ea5" containerName="revoke-keys"
Sep 16 12:15:40.877140 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.877121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:40.880132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.880112    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ad35c8a-openshift-default-dockercfg-gvjf8\""
Sep 16 12:15:40.893511 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.893487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"]
Sep 16 12:15:40.903790 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:40.903764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:41.062035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.061954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-data\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.061999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-certs\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.062022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-podinfo\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.062041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvff\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-kube-api-access-qcvff\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.062095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e22fb35-c7a3-462e-a910-1b66f07c460a-istiod-ca-cert\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.062143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-envoy\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.062165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-socket\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.062193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-credential-socket\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.062251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.062210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-token\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.108657 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.108615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:41.111606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.111556    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36da08cf-d986-4a4a-bb81-8b7a83c32ea5" path="/var/lib/kubelet/pods/36da08cf-d986-4a4a-bb81-8b7a83c32ea5/volumes"
Sep 16 12:15:41.111956 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.111943    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5116c5b0-260e-4333-818b-ec2553c370b5" path="/var/lib/kubelet/pods/5116c5b0-260e-4333-818b-ec2553c370b5/volumes"
Sep 16 12:15:41.112250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.112239    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a" path="/var/lib/kubelet/pods/a1a6b5a5-96e8-4a93-9abd-c8e0b11e931a/volumes"
Sep 16 12:15:41.162935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.162905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-data\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.162948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-certs\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.162967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-podinfo\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.162992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvff\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-kube-api-access-qcvff\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e22fb35-c7a3-462e-a910-1b66f07c460a-istiod-ca-cert\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-envoy\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-socket\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-credential-socket\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-token\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-data\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-certs\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163745 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-socket\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-credential-socket\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.163963 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.163942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e22fb35-c7a3-462e-a910-1b66f07c460a-istiod-ca-cert\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.165426 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.165407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-envoy\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.165556 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.165542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-podinfo\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.171506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.171478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvff\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-kube-api-access-qcvff\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.171758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.171741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-token\") pod \"e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") " pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.187447 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.187416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:41.209225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.209196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:41.466496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.466466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"]
Sep 16 12:15:41.467932 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:41.467907    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e22fb35_c7a3_462e_a910_1b66f07c460a.slice/crio-af206be81f035fb438c7492f604c9429d9249ebd3830a3049aae02bf78b1acb3 WatchSource:0}: Error finding container af206be81f035fb438c7492f604c9429d9249ebd3830a3049aae02bf78b1acb3: Status 404 returned error can't find the container with id af206be81f035fb438c7492f604c9429d9249ebd3830a3049aae02bf78b1acb3
Sep 16 12:15:41.470313 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.470283    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:41.470416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.470346    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:41.470416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.470374    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:15:41.776963 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.776876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:41.800402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.800363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" event={"ID":"7e22fb35-c7a3-462e-a910-1b66f07c460a","Type":"ContainerStarted","Data":"6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35"}
Sep 16 12:15:41.800619 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.800434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" event={"ID":"7e22fb35-c7a3-462e-a910-1b66f07c460a","Type":"ContainerStarted","Data":"af206be81f035fb438c7492f604c9429d9249ebd3830a3049aae02bf78b1acb3"}
Sep 16 12:15:41.820710 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:41.820659    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podStartSLOduration=1.820641365 podStartE2EDuration="1.820641365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:41.819268348 +0000 UTC m=+829.231285254" watchObservedRunningTime="2026-09-16 12:15:41.820641365 +0000 UTC m=+829.232658258"
Sep 16 12:15:42.188447 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:42.188408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:42.193642 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:42.193614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:42.480782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:42.480706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:42.808271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:42.808178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:42.809167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:42.809144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:15:43.181105 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:43.181069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:43.280183 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:43.280143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:43.987364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:43.987333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:15:43.989397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:43.989374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:44.892203 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:44.892171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:44.991627 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:44.991563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:45.340586 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.340541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:45.365599 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.365555    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:45.400952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.400859    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-data\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.400952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.400909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-podinfo\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.400952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.400934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-envoy\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.401225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.400985    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-certs\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.401225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.401009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-credential-socket\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.401225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.401039    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-socket\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.401225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.401091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istiod-ca-cert\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.401225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.401114    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdjhl\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-kube-api-access-sdjhl\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.401225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.401141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-token\") pod \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\" (UID: \"7471ed9a-f5fe-4646-812e-89db3f29a2c0\") "
Sep 16 12:15:45.401941 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.401870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:45.401941 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.401892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:45.402172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.402011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:45.402347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.402309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:45.402347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.402312    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:15:45.404193 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.404147    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:45.404455 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.404432    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-kube-api-access-sdjhl" (OuterVolumeSpecName: "kube-api-access-sdjhl") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "kube-api-access-sdjhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:15:45.404633 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.404612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:15:45.404858 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.404837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7471ed9a-f5fe-4646-812e-89db3f29a2c0" (UID: "7471ed9a-f5fe-4646-812e-89db3f29a2c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:15:45.442394 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.442353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:45.501947 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.501915    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.501947 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.501943    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.502143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.501960    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.502143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.501969    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.502143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.501978    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.502143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.501986    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7471ed9a-f5fe-4646-812e-89db3f29a2c0-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.502143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.501995    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.502143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.502003    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdjhl\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-kube-api-access-sdjhl\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.502143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.502012    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7471ed9a-f5fe-4646-812e-89db3f29a2c0-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:15:45.534050 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.534008    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"]
Sep 16 12:15:45.534442 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.534424    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy"
Sep 16 12:15:45.534529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.534444    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy"
Sep 16 12:15:45.534529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.534470    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy"
Sep 16 12:15:45.534529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.534478    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy"
Sep 16 12:15:45.534702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.534616    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy"
Sep 16 12:15:45.534702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.534632    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerName="istio-proxy"
Sep 16 12:15:45.536754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.536732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.538117 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.538084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:45.539658 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.539635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3ad35c8a-dockercfg-hjrjw\""
Sep 16 12:15:45.539781 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.539736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3ad35c8a\""
Sep 16 12:15:45.551205 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.551182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"]
Sep 16 12:15:45.571463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.571424    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"]
Sep 16 12:15:45.573797 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.573776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:45.584397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.584375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"]
Sep 16 12:15:45.603107 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.603078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8931a4d-4318-4900-876d-f57b266bc364-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.603292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.603118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gph5\" (UniqueName: \"kubernetes.io/projected/2b81995a-8c23-452d-920f-9b32e1135412-kube-api-access-6gph5\") pod \"payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") " pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:45.603292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.603139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81995a-8c23-452d-920f-9b32e1135412-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") " pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:45.603292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.603253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/b8931a4d-4318-4900-876d-f57b266bc364-kube-api-access-dl2z6\") pod \"payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.703974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.703866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/b8931a4d-4318-4900-876d-f57b266bc364-kube-api-access-dl2z6\") pod \"payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.703974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.703947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8931a4d-4318-4900-876d-f57b266bc364-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.703974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.703979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gph5\" (UniqueName: \"kubernetes.io/projected/2b81995a-8c23-452d-920f-9b32e1135412-kube-api-access-6gph5\") pod \"payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") " pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:45.704271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.703997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81995a-8c23-452d-920f-9b32e1135412-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") " pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:45.704750 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.704731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81995a-8c23-452d-920f-9b32e1135412-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") " pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:45.704801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.704751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8931a4d-4318-4900-876d-f57b266bc364-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.714002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.713964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gph5\" (UniqueName: \"kubernetes.io/projected/2b81995a-8c23-452d-920f-9b32e1135412-kube-api-access-6gph5\") pod \"payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") " pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:45.714002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.713963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/b8931a4d-4318-4900-876d-f57b266bc364-kube-api-access-dl2z6\") pod \"payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.784091 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.784046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:45.826203 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.826167    2577 generic.go:356] "Generic (PLEG): container finished" podID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" containerID="8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342" exitCode=0
Sep 16 12:15:45.826410 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.826214    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"
Sep 16 12:15:45.826410 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.826223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" event={"ID":"7471ed9a-f5fe-4646-812e-89db3f29a2c0","Type":"ContainerDied","Data":"8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342"}
Sep 16 12:15:45.826410 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.826271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn" event={"ID":"7471ed9a-f5fe-4646-812e-89db3f29a2c0","Type":"ContainerDied","Data":"bba09a5d3292d38fa59d712bbab9bcb6bcdebd3448a4d2d7505360d3859085ed"}
Sep 16 12:15:45.826410 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.826294    2577 scope.go:117] "RemoveContainer" containerID="8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342"
Sep 16 12:15:45.836447 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.836427    2577 scope.go:117] "RemoveContainer" containerID="73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a"
Sep 16 12:15:45.844863 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.844839    2577 scope.go:117] "RemoveContainer" containerID="8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342"
Sep 16 12:15:45.845143 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:15:45.845127    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342\": container with ID starting with 8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342 not found: ID does not exist" containerID="8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342"
Sep 16 12:15:45.845189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.845153    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342"} err="failed to get container status \"8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342\": rpc error: code = NotFound desc = could not find container \"8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342\": container with ID starting with 8a93fad97c30840845147d77159d512f07829fc95b0d56c66abfa595b4821342 not found: ID does not exist"
Sep 16 12:15:45.845189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.845171    2577 scope.go:117] "RemoveContainer" containerID="73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a"
Sep 16 12:15:45.845404 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:15:45.845387    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a\": container with ID starting with 73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a not found: ID does not exist" containerID="73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a"
Sep 16 12:15:45.845441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.845408    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a"} err="failed to get container status \"73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a\": rpc error: code = NotFound desc = could not find container \"73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a\": container with ID starting with 73cfe5a8d739c1383ec067d2bb8c09f6bda112dfe71172dbbbf93489ed3ed67a not found: ID does not exist"
Sep 16 12:15:45.849073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.848862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:45.849417 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.849355    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"]
Sep 16 12:15:45.859243 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.859177    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f74dc7e1-gw-openshift-default-7c59994fb4-btqpn"]
Sep 16 12:15:45.884615 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:45.884533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:46.137595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.132504    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"]
Sep 16 12:15:46.164544 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.164513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"]
Sep 16 12:15:46.166182 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:15:46.166155    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b81995a_8c23_452d_920f_9b32e1135412.slice/crio-402dfce7d0d40b3fefe53526c411920cd44903f0de44e67104e83377eff0eb2e WatchSource:0}: Error finding container 402dfce7d0d40b3fefe53526c411920cd44903f0de44e67104e83377eff0eb2e: Status 404 returned error can't find the container with id 402dfce7d0d40b3fefe53526c411920cd44903f0de44e67104e83377eff0eb2e
Sep 16 12:15:46.370973 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.370933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:46.834942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.834902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s" event={"ID":"2b81995a-8c23-452d-920f-9b32e1135412","Type":"ContainerStarted","Data":"c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f"}
Sep 16 12:15:46.835165 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.834978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:15:46.835165 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.835004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s" event={"ID":"2b81995a-8c23-452d-920f-9b32e1135412","Type":"ContainerStarted","Data":"402dfce7d0d40b3fefe53526c411920cd44903f0de44e67104e83377eff0eb2e"}
Sep 16 12:15:46.836710 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.836679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl" event={"ID":"b8931a4d-4318-4900-876d-f57b266bc364","Type":"ContainerStarted","Data":"ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4"}
Sep 16 12:15:46.836849 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.836732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl" event={"ID":"b8931a4d-4318-4900-876d-f57b266bc364","Type":"ContainerStarted","Data":"eac9d4da167422509890857384c8c12730689c1e8d6432ac6d34abf036b62045"}
Sep 16 12:15:46.836849 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.836795    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:46.853213 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.853162    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s" podStartSLOduration=1.853144197 podStartE2EDuration="1.853144197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:46.851136428 +0000 UTC m=+834.263153327" watchObservedRunningTime="2026-09-16 12:15:46.853144197 +0000 UTC m=+834.265161097"
Sep 16 12:15:46.866046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:46.866007    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl" podStartSLOduration=1.86599402 podStartE2EDuration="1.86599402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:46.865180616 +0000 UTC m=+834.277197512" watchObservedRunningTime="2026-09-16 12:15:46.86599402 +0000 UTC m=+834.278010916"
Sep 16 12:15:47.113351 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:47.113272    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7471ed9a-f5fe-4646-812e-89db3f29a2c0" path="/var/lib/kubelet/pods/7471ed9a-f5fe-4646-812e-89db3f29a2c0/volumes"
Sep 16 12:15:47.127221 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:47.127192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:47.818423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:47.818390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:48.238464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:48.238433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:15:57.849801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:57.849761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:15:57.850292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:15:57.849868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:16:04.324688 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:04.324645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:16:04.937986 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:04.937951    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerID="6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35" exitCode=0
Sep 16 12:16:04.938160 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:04.938028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" event={"ID":"7e22fb35-c7a3-462e-a910-1b66f07c460a","Type":"ContainerDied","Data":"6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35"}
Sep 16 12:16:04.938449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:04.938432    2577 scope.go:117] "RemoveContainer" containerID="6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35"
Sep 16 12:16:04.939760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:04.939731    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:04.939848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:04.939800    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:04.939848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:04.939833    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:05.945442 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:05.945406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" event={"ID":"7e22fb35-c7a3-462e-a910-1b66f07c460a","Type":"ContainerStarted","Data":"bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a"}
Sep 16 12:16:06.188309 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:06.188266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:16:06.188481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:06.188461    2577 patch_prober.go:28] interesting pod/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:16:06.188527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:06.188509    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 12:16:06.877579 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:06.877530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:07.188460 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:07.188420    2577 patch_prober.go:28] interesting pod/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:16:07.188870 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:07.188481    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 12:16:07.468142 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:07.468060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:08.087559 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:08.087525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:08.187913 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:08.187871    2577 patch_prober.go:28] interesting pod/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:16:08.188096 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:08.187943    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 12:16:08.721715 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:08.721679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:09.020335 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.020254    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"]
Sep 16 12:16:09.022692 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.022675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"
Sep 16 12:16:09.025220 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.025193    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:16:09.025330 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.025251    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:16:09.031481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.031448    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"]
Sep 16 12:16:09.122781 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.122748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m\" (UID: \"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"
Sep 16 12:16:09.188882 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.188845    2577 patch_prober.go:28] interesting pod/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:16:09.189045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.188920    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 12:16:09.224078 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.224041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m\" (UID: \"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"
Sep 16 12:16:09.224736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.224706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m\" (UID: \"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"
Sep 16 12:16:09.338251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.338139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"
Sep 16 12:16:09.548523 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.548441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:09.618812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.618201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"]
Sep 16 12:16:09.621070 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:16:09.621042    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca4f4d2_96bf_49eb_93ab_e6f8a9150e48.slice/crio-8effa9f05675876da5980ec13d9a2626480dec491c6ad49d1f2c2a6de2232038 WatchSource:0}: Error finding container 8effa9f05675876da5980ec13d9a2626480dec491c6ad49d1f2c2a6de2232038: Status 404 returned error can't find the container with id 8effa9f05675876da5980ec13d9a2626480dec491c6ad49d1f2c2a6de2232038
Sep 16 12:16:09.970028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.970001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m" event={"ID":"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48","Type":"ContainerStarted","Data":"26b86add5fc79d1c40db3600951f3783f0bd604cb5b9d3e4484460706143a580"}
Sep 16 12:16:09.970364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.970051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m" event={"ID":"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48","Type":"ContainerStarted","Data":"8effa9f05675876da5980ec13d9a2626480dec491c6ad49d1f2c2a6de2232038"}
Sep 16 12:16:09.987424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:09.987371    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m" podStartSLOduration=0.987356205 podStartE2EDuration="987.356205ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:09.985026481 +0000 UTC m=+857.397043377" watchObservedRunningTime="2026-09-16 12:16:09.987356205 +0000 UTC m=+857.399373098"
Sep 16 12:16:10.172064 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:10.172018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:10.188168 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:10.188140    2577 patch_prober.go:28] interesting pod/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:16:10.188324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:10.188190    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 12:16:10.399201 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:10.399157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:10.977847 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:10.977754    2577 generic.go:356] "Generic (PLEG): container finished" podID="0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48" containerID="26b86add5fc79d1c40db3600951f3783f0bd604cb5b9d3e4484460706143a580" exitCode=0
Sep 16 12:16:10.977847 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:10.977814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m" event={"ID":"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48","Type":"ContainerDied","Data":"26b86add5fc79d1c40db3600951f3783f0bd604cb5b9d3e4484460706143a580"}
Sep 16 12:16:10.980462 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:10.980440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:11.188873 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:11.188839    2577 patch_prober.go:28] interesting pod/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 12:16:11.189034 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:11.188901    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 12:16:11.661643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:11.661602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:12.108488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.108464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"
Sep 16 12:16:12.192353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.192325    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:16:12.192635 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.192606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:16:12.193371 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.193348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:16:12.272930 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.272846    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48-maas-api-service-ca\") pod \"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48\" (UID: \"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48\") "
Sep 16 12:16:12.273278 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.273243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48" (UID: "0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:16:12.273453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.273433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:12.374312 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.374273    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48-maas-api-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:12.874592 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.874537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:12.989640 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.989607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m" event={"ID":"0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48","Type":"ContainerDied","Data":"8effa9f05675876da5980ec13d9a2626480dec491c6ad49d1f2c2a6de2232038"}
Sep 16 12:16:12.989640 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.989629    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"
Sep 16 12:16:12.989840 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:12.989654    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8effa9f05675876da5980ec13d9a2626480dec491c6ad49d1f2c2a6de2232038"
Sep 16 12:16:13.565895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:13.565856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:14.487799 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:14.487763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.010714 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.010681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.036844 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.036807    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"]
Sep 16 12:16:15.037061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.037032    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s" podUID="2b81995a-8c23-452d-920f-9b32e1135412" containerName="payload-processing" containerID="cri-o://c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f" gracePeriod=30
Sep 16 12:16:15.045550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.045181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"]
Sep 16 12:16:15.045550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.045408    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl" podUID="b8931a4d-4318-4900-876d-f57b266bc364" containerName="payload-pre-processing" containerID="cri-o://ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4" gracePeriod=30
Sep 16 12:16:15.076000 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.075964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.175013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.174951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.209248 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.209204    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"]
Sep 16 12:16:15.211473 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.211199    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ad35c8a-f2t5m"]
Sep 16 12:16:15.234582 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.234540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.338764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.338733    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:16:15.342107 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.342084    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:16:15.496026 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.495985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:15.501881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.501858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/b8931a4d-4318-4900-876d-f57b266bc364-kube-api-access-dl2z6\") pod \"b8931a4d-4318-4900-876d-f57b266bc364\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") "
Sep 16 12:16:15.501989 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.501904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81995a-8c23-452d-920f-9b32e1135412-plugins-config-volume\") pod \"2b81995a-8c23-452d-920f-9b32e1135412\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") "
Sep 16 12:16:15.501989 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.501937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gph5\" (UniqueName: \"kubernetes.io/projected/2b81995a-8c23-452d-920f-9b32e1135412-kube-api-access-6gph5\") pod \"2b81995a-8c23-452d-920f-9b32e1135412\" (UID: \"2b81995a-8c23-452d-920f-9b32e1135412\") "
Sep 16 12:16:15.501989 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.501964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8931a4d-4318-4900-876d-f57b266bc364-plugins-config-volume\") pod \"b8931a4d-4318-4900-876d-f57b266bc364\" (UID: \"b8931a4d-4318-4900-876d-f57b266bc364\") "
Sep 16 12:16:15.502350 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.502328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b81995a-8c23-452d-920f-9b32e1135412-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b81995a-8c23-452d-920f-9b32e1135412" (UID: "2b81995a-8c23-452d-920f-9b32e1135412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:16:15.502407 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.502345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8931a4d-4318-4900-876d-f57b266bc364-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8931a4d-4318-4900-876d-f57b266bc364" (UID: "b8931a4d-4318-4900-876d-f57b266bc364"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:16:15.504103 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.504068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8931a4d-4318-4900-876d-f57b266bc364-kube-api-access-dl2z6" (OuterVolumeSpecName: "kube-api-access-dl2z6") pod "b8931a4d-4318-4900-876d-f57b266bc364" (UID: "b8931a4d-4318-4900-876d-f57b266bc364"). InnerVolumeSpecName "kube-api-access-dl2z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:15.504238 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.504142    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b81995a-8c23-452d-920f-9b32e1135412-kube-api-access-6gph5" (OuterVolumeSpecName: "kube-api-access-6gph5") pod "2b81995a-8c23-452d-920f-9b32e1135412" (UID: "2b81995a-8c23-452d-920f-9b32e1135412"). InnerVolumeSpecName "kube-api-access-6gph5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:15.603161 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.603075    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2z6\" (UniqueName: \"kubernetes.io/projected/b8931a4d-4318-4900-876d-f57b266bc364-kube-api-access-dl2z6\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:15.603161 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.603108    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81995a-8c23-452d-920f-9b32e1135412-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:15.603161 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.603117    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gph5\" (UniqueName: \"kubernetes.io/projected/2b81995a-8c23-452d-920f-9b32e1135412-kube-api-access-6gph5\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:15.603161 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:15.603127    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8931a4d-4318-4900-876d-f57b266bc364-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:16.006091 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.006050    2577 generic.go:356] "Generic (PLEG): container finished" podID="2b81995a-8c23-452d-920f-9b32e1135412" containerID="c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f" exitCode=0
Sep 16 12:16:16.006287 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.006110    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"
Sep 16 12:16:16.006287 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.006159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s" event={"ID":"2b81995a-8c23-452d-920f-9b32e1135412","Type":"ContainerDied","Data":"c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f"}
Sep 16 12:16:16.006287 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.006262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s" event={"ID":"2b81995a-8c23-452d-920f-9b32e1135412","Type":"ContainerDied","Data":"402dfce7d0d40b3fefe53526c411920cd44903f0de44e67104e83377eff0eb2e"}
Sep 16 12:16:16.006441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.006304    2577 scope.go:117] "RemoveContainer" containerID="c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f"
Sep 16 12:16:16.007654 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.007635    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8931a4d-4318-4900-876d-f57b266bc364" containerID="ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4" exitCode=0
Sep 16 12:16:16.007760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.007693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"
Sep 16 12:16:16.007760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.007733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl" event={"ID":"b8931a4d-4318-4900-876d-f57b266bc364","Type":"ContainerDied","Data":"ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4"}
Sep 16 12:16:16.007839 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.007783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl" event={"ID":"b8931a4d-4318-4900-876d-f57b266bc364","Type":"ContainerDied","Data":"eac9d4da167422509890857384c8c12730689c1e8d6432ac6d34abf036b62045"}
Sep 16 12:16:16.016314 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.016299    2577 scope.go:117] "RemoveContainer" containerID="c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f"
Sep 16 12:16:16.016670 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:16:16.016608    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f\": container with ID starting with c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f not found: ID does not exist" containerID="c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f"
Sep 16 12:16:16.016670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.016632    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f"} err="failed to get container status \"c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f\": rpc error: code = NotFound desc = could not find container \"c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f\": container with ID starting with c83694870930e1408504d272ea16ba9bc302cdbd0034c40874f26b9ab8c5b84f not found: ID does not exist"
Sep 16 12:16:16.016670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.016648    2577 scope.go:117] "RemoveContainer" containerID="ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4"
Sep 16 12:16:16.024498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.024475    2577 scope.go:117] "RemoveContainer" containerID="ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4"
Sep 16 12:16:16.024768 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:16:16.024750    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4\": container with ID starting with ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4 not found: ID does not exist" containerID="ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4"
Sep 16 12:16:16.024844 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.024778    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4"} err="failed to get container status \"ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4\": rpc error: code = NotFound desc = could not find container \"ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4\": container with ID starting with ae9c3627b80e30c8224fd25e43b0bd17cffa17cf621ddb7bb65c447ae09dc7b4 not found: ID does not exist"
Sep 16 12:16:16.030849 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.030826    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"]
Sep 16 12:16:16.037001 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.036981    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ad35c8a-78f56b7cf-24n5s"]
Sep 16 12:16:16.048184 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.048162    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"]
Sep 16 12:16:16.051805 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.051783    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ad35c8a-6d79844594-r6wbl"]
Sep 16 12:16:16.146221 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.146189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:16.358286 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.358206    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"]
Sep 16 12:16:16.358488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.358462    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy" containerID="cri-o://bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a" gracePeriod=30
Sep 16 12:16:16.372234 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.372205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:16.872314 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.872284    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:16.971820 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:16.971788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:17.114471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.114434    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48" path="/var/lib/kubelet/pods/0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48/volumes"
Sep 16 12:16:17.114993 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.114970    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b81995a-8c23-452d-920f-9b32e1135412" path="/var/lib/kubelet/pods/2b81995a-8c23-452d-920f-9b32e1135412/volumes"
Sep 16 12:16:17.115402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.115379    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8931a4d-4318-4900-876d-f57b266bc364" path="/var/lib/kubelet/pods/b8931a4d-4318-4900-876d-f57b266bc364/volumes"
Sep 16 12:16:17.126800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.126728    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"]
Sep 16 12:16:17.127114 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127100    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8931a4d-4318-4900-876d-f57b266bc364" containerName="payload-pre-processing"
Sep 16 12:16:17.127199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127116    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8931a4d-4318-4900-876d-f57b266bc364" containerName="payload-pre-processing"
Sep 16 12:16:17.127199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127155    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48" containerName="revoke-keys"
Sep 16 12:16:17.127199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127162    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48" containerName="revoke-keys"
Sep 16 12:16:17.127199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127172    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b81995a-8c23-452d-920f-9b32e1135412" containerName="payload-processing"
Sep 16 12:16:17.127199 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127177    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b81995a-8c23-452d-920f-9b32e1135412" containerName="payload-processing"
Sep 16 12:16:17.127464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127256    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b81995a-8c23-452d-920f-9b32e1135412" containerName="payload-processing"
Sep 16 12:16:17.127464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127266    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8931a4d-4318-4900-876d-f57b266bc364" containerName="payload-pre-processing"
Sep 16 12:16:17.127464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.127273    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca4f4d2-96bf-49eb-93ab-e6f8a9150e48" containerName="revoke-keys"
Sep 16 12:16:17.131481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.131459    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.134106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.134086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a1d14a87-openshift-default-dockercfg-9hbf6\""
Sep 16 12:16:17.141884 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.141864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"]
Sep 16 12:16:17.168835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.168808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:17.318071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-socket\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-envoy\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81701c23-2746-4bab-907b-cf7930434a6d-istiod-ca-cert\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-certs\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5x86\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-kube-api-access-b5x86\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-istio-token\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-data\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81701c23-2746-4bab-907b-cf7930434a6d-istio-podinfo\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.318683 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.318386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-credential-socket\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.343657 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.343603    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:17.419416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-data\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81701c23-2746-4bab-907b-cf7930434a6d-istio-podinfo\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-credential-socket\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-socket\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-envoy\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81701c23-2746-4bab-907b-cf7930434a6d-istiod-ca-cert\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-certs\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x86\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-kube-api-access-b5x86\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-istio-token\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-data\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.419985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.419947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-credential-socket\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.420391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.420351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-certs\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.420598 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.420461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-socket\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.420598 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.420511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81701c23-2746-4bab-907b-cf7930434a6d-istiod-ca-cert\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.422324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.422300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-envoy\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.422643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.422623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81701c23-2746-4bab-907b-cf7930434a6d-istio-podinfo\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.428080 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.428034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-istio-token\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.428241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.428194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x86\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-kube-api-access-b5x86\") pod \"e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") " pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.443933 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.443902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:17.727099 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.727069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"]
Sep 16 12:16:17.727883 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:16:17.727844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81701c23_2746_4bab_907b_cf7930434a6d.slice/crio-cb44ccf5869af3c6a9f3ad907351e6bd027bc932a0ac69863f3a806eb91b4b74 WatchSource:0}: Error finding container cb44ccf5869af3c6a9f3ad907351e6bd027bc932a0ac69863f3a806eb91b4b74: Status 404 returned error can't find the container with id cb44ccf5869af3c6a9f3ad907351e6bd027bc932a0ac69863f3a806eb91b4b74
Sep 16 12:16:17.730484 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.730454    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:17.730560 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.730514    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:17.730624 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.730564    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:17.754643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:17.754610    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:18.021775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:18.021677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" event={"ID":"81701c23-2746-4bab-907b-cf7930434a6d","Type":"ContainerStarted","Data":"f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30"}
Sep 16 12:16:18.021775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:18.021729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" event={"ID":"81701c23-2746-4bab-907b-cf7930434a6d","Type":"ContainerStarted","Data":"cb44ccf5869af3c6a9f3ad907351e6bd027bc932a0ac69863f3a806eb91b4b74"}
Sep 16 12:16:18.044765 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:18.044714    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" podStartSLOduration=1.044699633 podStartE2EDuration="1.044699633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:18.042507538 +0000 UTC m=+865.454524435" watchObservedRunningTime="2026-09-16 12:16:18.044699633 +0000 UTC m=+865.456716528"
Sep 16 12:16:18.444656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:18.444614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:18.449800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:18.449776    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:18.751783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:18.751698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:19.028458 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:19.028370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:19.029464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:19.029438    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:19.383766 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:19.383730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:19.481817 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:19.481780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:20.211761 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:20.211723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:20.818540 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:20.818503    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:21.585226 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.585185    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:21.623749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.623680    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:16:21.661729 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-socket\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661729 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-data\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-certs\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e22fb35-c7a3-462e-a910-1b66f07c460a-istiod-ca-cert\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-credential-socket\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-podinfo\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661901    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-envoy\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcvff\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-kube-api-access-qcvff\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.661990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661967    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-token\") pod \"7e22fb35-c7a3-462e-a910-1b66f07c460a\" (UID: \"7e22fb35-c7a3-462e-a910-1b66f07c460a\") "
Sep 16 12:16:21.662353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.661975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:16:21.662353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.662175    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e22fb35-c7a3-462e-a910-1b66f07c460a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:16:21.662353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.662226    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.662353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.662298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:16:21.662664 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.662638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:16:21.662914 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.662891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:16:21.664397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.664359    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-kube-api-access-qcvff" (OuterVolumeSpecName: "kube-api-access-qcvff") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "kube-api-access-qcvff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:21.664497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.664391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:16:21.664497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.664442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:16:21.664917 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.664898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e22fb35-c7a3-462e-a910-1b66f07c460a" (UID: "7e22fb35-c7a3-462e-a910-1b66f07c460a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:16:21.763235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763190    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e22fb35-c7a3-462e-a910-1b66f07c460a-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.763235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763228    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.763235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763240    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.763235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763251    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.763498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763261    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvff\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-kube-api-access-qcvff\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.763498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763271    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.763498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763279    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:21.763498 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:21.763287    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e22fb35-c7a3-462e-a910-1b66f07c460a-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:16:22.046314 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.046276    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerID="bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a" exitCode=0
Sep 16 12:16:22.046719 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.046321    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"
Sep 16 12:16:22.046719 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.046351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" event={"ID":"7e22fb35-c7a3-462e-a910-1b66f07c460a","Type":"ContainerDied","Data":"bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a"}
Sep 16 12:16:22.046719 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.046409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr" event={"ID":"7e22fb35-c7a3-462e-a910-1b66f07c460a","Type":"ContainerDied","Data":"af206be81f035fb438c7492f604c9429d9249ebd3830a3049aae02bf78b1acb3"}
Sep 16 12:16:22.046719 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.046441    2577 scope.go:117] "RemoveContainer" containerID="bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a"
Sep 16 12:16:22.055903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.055883    2577 scope.go:117] "RemoveContainer" containerID="6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35"
Sep 16 12:16:22.066155 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.066135    2577 scope.go:117] "RemoveContainer" containerID="bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a"
Sep 16 12:16:22.066524 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:16:22.066482    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a\": container with ID starting with bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a not found: ID does not exist" containerID="bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a"
Sep 16 12:16:22.066635 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.066517    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a"} err="failed to get container status \"bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a\": rpc error: code = NotFound desc = could not find container \"bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a\": container with ID starting with bedfc40901e88f59b2cac4e2aea69c6caca2a1d6e21f94b33b58dc106728354a not found: ID does not exist"
Sep 16 12:16:22.066635 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.066540    2577 scope.go:117] "RemoveContainer" containerID="6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35"
Sep 16 12:16:22.066903 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:16:22.066880    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35\": container with ID starting with 6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35 not found: ID does not exist" containerID="6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35"
Sep 16 12:16:22.066962 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.066912    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35"} err="failed to get container status \"6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35\": rpc error: code = NotFound desc = could not find container \"6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35\": container with ID starting with 6175eadd560b6625060d310367d993e0998c88f39266127d293c3783e519cb35 not found: ID does not exist"
Sep 16 12:16:22.068328 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.068308    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"]
Sep 16 12:16:22.071352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.071330    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ad35c8a-openshift-default-56986cf9d4-ptqlr"]
Sep 16 12:16:22.182153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.182119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:22.767312 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:22.767274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:23.111791 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.111704    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" path="/var/lib/kubelet/pods/7e22fb35-c7a3-462e-a910-1b66f07c460a/volumes"
Sep 16 12:16:23.488938 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.488903    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"]
Sep 16 12:16:23.489420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.489397    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy"
Sep 16 12:16:23.489420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.489422    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy"
Sep 16 12:16:23.489634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.489534    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy"
Sep 16 12:16:23.489634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.489546    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy"
Sep 16 12:16:23.494110 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.494090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.500373 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.500351    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:16:23.500499 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.500438    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:16:23.500597 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.500509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:16:23.500597 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.500351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a1d14a87-serving-cert\""
Sep 16 12:16:23.501319 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.501110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7fbn\""
Sep 16 12:16:23.511918 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.511887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"]
Sep 16 12:16:23.539492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.539455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:23.576857 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.576819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.576857 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.576859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-metrics-certs\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.577068 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.576898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c557fe-04cc-4bde-ac84-62bfe1ea249f-openshift-service-ca\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.577068 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.577019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2glg\" (UniqueName: \"kubernetes.io/projected/49c557fe-04cc-4bde-ac84-62bfe1ea249f-kube-api-access-g2glg\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.620323 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.620210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:23.678430 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.678384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.678430 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.678436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-metrics-certs\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.678722 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.678488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c557fe-04cc-4bde-ac84-62bfe1ea249f-openshift-service-ca\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.678722 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.678535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2glg\" (UniqueName: \"kubernetes.io/projected/49c557fe-04cc-4bde-ac84-62bfe1ea249f-kube-api-access-g2glg\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.678722 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:16:23.678542    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-serving-cert: secret "maas-api-e2e-derive-a1d14a87-serving-cert" not found
Sep 16 12:16:23.678722 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:16:23.678648    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls podName:49c557fe-04cc-4bde-ac84-62bfe1ea249f nodeName:}" failed. No retries permitted until 2026-09-16 12:16:24.178627382 +0000 UTC m=+871.590644276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls") pod "maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2" (UID: "49c557fe-04cc-4bde-ac84-62bfe1ea249f") : secret "maas-api-e2e-derive-a1d14a87-serving-cert" not found
Sep 16 12:16:23.679235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.679215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c557fe-04cc-4bde-ac84-62bfe1ea249f-openshift-service-ca\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.680920 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.680897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-metrics-certs\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.689467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.689442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2glg\" (UniqueName: \"kubernetes.io/projected/49c557fe-04cc-4bde-ac84-62bfe1ea249f-kube-api-access-g2glg\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:23.839776 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.839683    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:23.964823 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:23.964779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:24.064266 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.064232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:24.073086 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.073057    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"]
Sep 16 12:16:24.073478 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.073461    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy"
Sep 16 12:16:24.073551 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.073482    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e22fb35-c7a3-462e-a910-1b66f07c460a" containerName="istio-proxy"
Sep 16 12:16:24.076761 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.076740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.079491 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.079467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a1d14a87\""
Sep 16 12:16:24.079607 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.079514    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a1d14a87-dockercfg-p7wcv\""
Sep 16 12:16:24.081171 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.081151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fd6c\" (UniqueName: \"kubernetes.io/projected/2565fe4e-20f5-4089-aa15-818b57d61d71-kube-api-access-9fd6c\") pod \"payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.081271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.081209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2565fe4e-20f5-4089-aa15-818b57d61d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.089155 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.089134    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"]
Sep 16 12:16:24.181755 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.181717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:24.182293 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.181768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fd6c\" (UniqueName: \"kubernetes.io/projected/2565fe4e-20f5-4089-aa15-818b57d61d71-kube-api-access-9fd6c\") pod \"payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.182293 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.181807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2565fe4e-20f5-4089-aa15-818b57d61d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.182517 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.182495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2565fe4e-20f5-4089-aa15-818b57d61d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.184926 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.184891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls\") pod \"maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:24.192661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.192632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fd6c\" (UniqueName: \"kubernetes.io/projected/2565fe4e-20f5-4089-aa15-818b57d61d71-kube-api-access-9fd6c\") pod \"payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.386678 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.386642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:24.406711 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.406685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:24.658808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.658783    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"]
Sep 16 12:16:24.660157 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:16:24.660127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2565fe4e_20f5_4089_aa15_818b57d61d71.slice/crio-01c0114ae1b7d241828c9f972070c7720a4b63d247988a5d6f2a5e385fa23186 WatchSource:0}: Error finding container 01c0114ae1b7d241828c9f972070c7720a4b63d247988a5d6f2a5e385fa23186: Status 404 returned error can't find the container with id 01c0114ae1b7d241828c9f972070c7720a4b63d247988a5d6f2a5e385fa23186
Sep 16 12:16:24.685376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:24.685353    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"]
Sep 16 12:16:24.689645 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:16:24.689619    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c557fe_04cc_4bde_ac84_62bfe1ea249f.slice/crio-40e33927a2a3334c821c4328e0305a1c0d94a5bfa2d5f303270d184eec54d008 WatchSource:0}: Error finding container 40e33927a2a3334c821c4328e0305a1c0d94a5bfa2d5f303270d184eec54d008: Status 404 returned error can't find the container with id 40e33927a2a3334c821c4328e0305a1c0d94a5bfa2d5f303270d184eec54d008
Sep 16 12:16:25.064824 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:25.064727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2" event={"ID":"49c557fe-04cc-4bde-ac84-62bfe1ea249f","Type":"ContainerStarted","Data":"40e33927a2a3334c821c4328e0305a1c0d94a5bfa2d5f303270d184eec54d008"}
Sep 16 12:16:25.066361 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:25.066330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht" event={"ID":"2565fe4e-20f5-4089-aa15-818b57d61d71","Type":"ContainerStarted","Data":"693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83"}
Sep 16 12:16:25.066506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:25.066372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht" event={"ID":"2565fe4e-20f5-4089-aa15-818b57d61d71","Type":"ContainerStarted","Data":"01c0114ae1b7d241828c9f972070c7720a4b63d247988a5d6f2a5e385fa23186"}
Sep 16 12:16:25.066506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:25.066439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:25.083725 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:25.083660    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht" podStartSLOduration=1.083641977 podStartE2EDuration="1.083641977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:25.081199004 +0000 UTC m=+872.493215899" watchObservedRunningTime="2026-09-16 12:16:25.083641977 +0000 UTC m=+872.495658873"
Sep 16 12:16:28.087139 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:28.087097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2" event={"ID":"49c557fe-04cc-4bde-ac84-62bfe1ea249f","Type":"ContainerStarted","Data":"781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26"}
Sep 16 12:16:28.087645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:28.087176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:28.105041 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:28.104986    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2" podStartSLOduration=2.626233858 podStartE2EDuration="5.104970774s" totalImagesPullingTime="2.478736916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:16:28.103194597 +0000 UTC m=+875.515211492" watchObservedRunningTime="2026-09-16 12:16:28.104970774 +0000 UTC m=+875.516987669"
Sep 16 12:16:30.784035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:30.783998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:16:34.097903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:34.097869    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:16:36.075463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:36.075430    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:16:37.964484 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:37.964452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:40.159680 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:40.159590    2577 generic.go:356] "Generic (PLEG): container finished" podID="81701c23-2746-4bab-907b-cf7930434a6d" containerID="f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30" exitCode=0
Sep 16 12:16:40.159680 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:40.159640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" event={"ID":"81701c23-2746-4bab-907b-cf7930434a6d","Type":"ContainerDied","Data":"f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30"}
Sep 16 12:16:40.160104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:40.160064    2577 scope.go:117] "RemoveContainer" containerID="f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30"
Sep 16 12:16:40.161269 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:40.161244    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:40.161327 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:40.161312    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:40.161365 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:40.161342    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:16:41.166731 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:41.166690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" event={"ID":"81701c23-2746-4bab-907b-cf7930434a6d","Type":"ContainerStarted","Data":"c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60"}
Sep 16 12:16:41.445519 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:41.445412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:41.445712 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:41.445545    2577 patch_prober.go:28] interesting pod/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:16:41.445712 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:41.445614    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 16 12:16:42.445406 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:42.445355    2577 patch_prober.go:28] interesting pod/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:16:42.445901 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:42.445429    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 16 12:16:43.445414 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:43.445372    2577 patch_prober.go:28] interesting pod/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:16:43.446051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:43.446017    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 16 12:16:44.445629 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:44.445583    2577 patch_prober.go:28] interesting pod/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:16:44.446147 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:44.445657    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 16 12:16:45.258223 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:45.258140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:45.445258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:45.445213    2577 patch_prober.go:28] interesting pod/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:16:45.445595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:45.445532    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 16 12:16:45.949813 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:45.949778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:46.451197 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:46.451037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:46.451999 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:46.451971    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:46.452805 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:46.452781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:16:46.574703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:46.574670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:46.867997 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:46.867904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:47.467150 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:47.467119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:48.183721 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:48.183690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:48.774718 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:48.774689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:48.889556 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:48.889525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:49.656105 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:49.656072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:50.270326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:50.270273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:51.024374 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:51.024334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:51.670291 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:51.670259    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:52.379957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:52.379919    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:52.981859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:52.981824    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:53.580477 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:53.580425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:54.204279 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:54.204225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:55.097013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:55.096970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:55.607365 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:55.607309    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"]
Sep 16 12:16:55.609534 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:55.609511    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-dc3e2c-29826015-6nrjc"]
Sep 16 12:16:55.837855 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:55.837809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:55.937088 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:55.937054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:56.150603 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:56.150550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:56.193506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:56.193425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:16:56.253704 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:56.253669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:57.089863 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:57.089821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:57.112799 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:57.112759    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="376911dd-a23c-4283-8aeb-05d30cd34cba" path="/var/lib/kubelet/pods/376911dd-a23c-4283-8aeb-05d30cd34cba/volumes"
Sep 16 12:16:57.191911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:57.191869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:57.542824 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:57.542787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:58.197937 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:58.197901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:58.805648 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:58.805610    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:16:59.383465 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:16:59.383429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:00.702726 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:00.702692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:01.201097 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.201064    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"]
Sep 16 12:17:01.201315 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.201293    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2" podUID="49c557fe-04cc-4bde-ac84-62bfe1ea249f" containerName="maas-api" containerID="cri-o://781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26" gracePeriod=30
Sep 16 12:17:01.203130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.203104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:01.248071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.248036    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"]
Sep 16 12:17:01.248295 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.248263    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht" podUID="2565fe4e-20f5-4089-aa15-818b57d61d71" containerName="payload-pre-processing" containerID="cri-o://693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83" gracePeriod=30
Sep 16 12:17:01.300339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.300298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:01.389267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.389236    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:01.451336 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.451238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:01.515090 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.515063    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:17:01.533667 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.533635    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:17:01.563055 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.563018    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"]
Sep 16 12:17:01.563305 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.563276    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy" containerID="cri-o://c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60" gracePeriod=30
Sep 16 12:17:01.580093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.580059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:01.628311 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c557fe-04cc-4bde-ac84-62bfe1ea249f-openshift-service-ca\") pod \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") "
Sep 16 12:17:01.628311 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2glg\" (UniqueName: \"kubernetes.io/projected/49c557fe-04cc-4bde-ac84-62bfe1ea249f-kube-api-access-g2glg\") pod \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") "
Sep 16 12:17:01.628703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls\") pod \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") "
Sep 16 12:17:01.628703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628390    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2565fe4e-20f5-4089-aa15-818b57d61d71-plugins-config-volume\") pod \"2565fe4e-20f5-4089-aa15-818b57d61d71\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") "
Sep 16 12:17:01.628703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-metrics-certs\") pod \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\" (UID: \"49c557fe-04cc-4bde-ac84-62bfe1ea249f\") "
Sep 16 12:17:01.628703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fd6c\" (UniqueName: \"kubernetes.io/projected/2565fe4e-20f5-4089-aa15-818b57d61d71-kube-api-access-9fd6c\") pod \"2565fe4e-20f5-4089-aa15-818b57d61d71\" (UID: \"2565fe4e-20f5-4089-aa15-818b57d61d71\") "
Sep 16 12:17:01.628911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628775    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c557fe-04cc-4bde-ac84-62bfe1ea249f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49c557fe-04cc-4bde-ac84-62bfe1ea249f" (UID: "49c557fe-04cc-4bde-ac84-62bfe1ea249f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:17:01.628911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.628879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2565fe4e-20f5-4089-aa15-818b57d61d71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2565fe4e-20f5-4089-aa15-818b57d61d71" (UID: "2565fe4e-20f5-4089-aa15-818b57d61d71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:17:01.630844 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.630813    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49c557fe-04cc-4bde-ac84-62bfe1ea249f" (UID: "49c557fe-04cc-4bde-ac84-62bfe1ea249f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:17:01.631068 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.631043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49c557fe-04cc-4bde-ac84-62bfe1ea249f" (UID: "49c557fe-04cc-4bde-ac84-62bfe1ea249f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:17:01.631123 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.631107    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c557fe-04cc-4bde-ac84-62bfe1ea249f-kube-api-access-g2glg" (OuterVolumeSpecName: "kube-api-access-g2glg") pod "49c557fe-04cc-4bde-ac84-62bfe1ea249f" (UID: "49c557fe-04cc-4bde-ac84-62bfe1ea249f"). InnerVolumeSpecName "kube-api-access-g2glg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:17:01.631288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.631266    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2565fe4e-20f5-4089-aa15-818b57d61d71-kube-api-access-9fd6c" (OuterVolumeSpecName: "kube-api-access-9fd6c") pod "2565fe4e-20f5-4089-aa15-818b57d61d71" (UID: "2565fe4e-20f5-4089-aa15-818b57d61d71"). InnerVolumeSpecName "kube-api-access-9fd6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:17:01.730039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.729950    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-metrics-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.730039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.729981    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fd6c\" (UniqueName: \"kubernetes.io/projected/2565fe4e-20f5-4089-aa15-818b57d61d71-kube-api-access-9fd6c\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.730039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.729992    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49c557fe-04cc-4bde-ac84-62bfe1ea249f-openshift-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.730039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.730001    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2glg\" (UniqueName: \"kubernetes.io/projected/49c557fe-04cc-4bde-ac84-62bfe1ea249f-kube-api-access-g2glg\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.730039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.730010    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49c557fe-04cc-4bde-ac84-62bfe1ea249f-maas-api-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.730039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.730018    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2565fe4e-20f5-4089-aa15-818b57d61d71-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:01.888029 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:01.887994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:02.284467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.284431    2577 generic.go:356] "Generic (PLEG): container finished" podID="49c557fe-04cc-4bde-ac84-62bfe1ea249f" containerID="781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26" exitCode=0
Sep 16 12:17:02.284673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.284480    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"
Sep 16 12:17:02.284673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.284518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2" event={"ID":"49c557fe-04cc-4bde-ac84-62bfe1ea249f","Type":"ContainerDied","Data":"781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26"}
Sep 16 12:17:02.284673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.284595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2" event={"ID":"49c557fe-04cc-4bde-ac84-62bfe1ea249f","Type":"ContainerDied","Data":"40e33927a2a3334c821c4328e0305a1c0d94a5bfa2d5f303270d184eec54d008"}
Sep 16 12:17:02.284673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.284623    2577 scope.go:117] "RemoveContainer" containerID="781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26"
Sep 16 12:17:02.286268 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.286241    2577 generic.go:356] "Generic (PLEG): container finished" podID="2565fe4e-20f5-4089-aa15-818b57d61d71" containerID="693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83" exitCode=0
Sep 16 12:17:02.286390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.286302    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"
Sep 16 12:17:02.286390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.286321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht" event={"ID":"2565fe4e-20f5-4089-aa15-818b57d61d71","Type":"ContainerDied","Data":"693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83"}
Sep 16 12:17:02.286390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.286373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht" event={"ID":"2565fe4e-20f5-4089-aa15-818b57d61d71","Type":"ContainerDied","Data":"01c0114ae1b7d241828c9f972070c7720a4b63d247988a5d6f2a5e385fa23186"}
Sep 16 12:17:02.294835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.294814    2577 scope.go:117] "RemoveContainer" containerID="781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26"
Sep 16 12:17:02.295117 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:02.295096    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26\": container with ID starting with 781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26 not found: ID does not exist" containerID="781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26"
Sep 16 12:17:02.295168 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.295128    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26"} err="failed to get container status \"781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26\": rpc error: code = NotFound desc = could not find container \"781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26\": container with ID starting with 781601965c4bdf4a7fa68b8e1c44f0b177d44b7272b997a46df11d8d6b7f5b26 not found: ID does not exist"
Sep 16 12:17:02.295168 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.295150    2577 scope.go:117] "RemoveContainer" containerID="693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83"
Sep 16 12:17:02.304312 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.304292    2577 scope.go:117] "RemoveContainer" containerID="693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83"
Sep 16 12:17:02.304595 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:02.304557    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83\": container with ID starting with 693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83 not found: ID does not exist" containerID="693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83"
Sep 16 12:17:02.304682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.304601    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83"} err="failed to get container status \"693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83\": rpc error: code = NotFound desc = could not find container \"693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83\": container with ID starting with 693fdd787bb77f885722b8e2e93f1ff292bce788ceba99822628683b45e65a83 not found: ID does not exist"
Sep 16 12:17:02.318000 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.317973    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"]
Sep 16 12:17:02.333227 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.333194    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1d14a87-569bf88b54-lk8p2"]
Sep 16 12:17:02.334448 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.334403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:02.344447 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.344423    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"]
Sep 16 12:17:02.349647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.349623    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a1d14a87-758966f5fd-tckht"]
Sep 16 12:17:02.427675 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.427641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:02.527397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.527310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:02.612264 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:02.612233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:03.111450 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:03.111413    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2565fe4e-20f5-4089-aa15-818b57d61d71" path="/var/lib/kubelet/pods/2565fe4e-20f5-4089-aa15-818b57d61d71/volumes"
Sep 16 12:17:03.111854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:03.111840    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c557fe-04cc-4bde-ac84-62bfe1ea249f" path="/var/lib/kubelet/pods/49c557fe-04cc-4bde-ac84-62bfe1ea249f/volumes"
Sep 16 12:17:03.503812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:03.503758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:03.996666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:03.996632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:04.178772 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:04.178733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:04.971478 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:04.971436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:06.821607 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.821583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:17:06.831757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.831693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:06.974750 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.974709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-socket\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.974750 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.974754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81701c23-2746-4bab-907b-cf7930434a6d-istiod-ca-cert\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.974777    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5x86\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-kube-api-access-b5x86\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.974795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-data\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.974813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-credential-socket\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.974944    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81701c23-2746-4bab-907b-cf7930434a6d-istio-podinfo\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-istio-token\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975083    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:06.975216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975134    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:06.975216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-envoy\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-certs\") pod \"81701c23-2746-4bab-907b-cf7930434a6d\" (UID: \"81701c23-2746-4bab-907b-cf7930434a6d\") "
Sep 16 12:17:06.975216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:06.975524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81701c23-2746-4bab-907b-cf7930434a6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:17:06.975524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975471    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:06.975524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975489    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81701c23-2746-4bab-907b-cf7930434a6d-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:06.975524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975502    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:06.975524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975517    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:06.975829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.975561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:06.977046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.977019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81701c23-2746-4bab-907b-cf7930434a6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:17:06.977152 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.977098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:17:06.977399 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.977367    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:06.977399 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:06.977372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-kube-api-access-b5x86" (OuterVolumeSpecName: "kube-api-access-b5x86") pod "81701c23-2746-4bab-907b-cf7930434a6d" (UID: "81701c23-2746-4bab-907b-cf7930434a6d"). InnerVolumeSpecName "kube-api-access-b5x86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:17:07.076531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.076494    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5x86\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-kube-api-access-b5x86\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:07.076531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.076535    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81701c23-2746-4bab-907b-cf7930434a6d-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:07.076754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.076551    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81701c23-2746-4bab-907b-cf7930434a6d-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:07.076754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.076563    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:07.076754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.076593    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81701c23-2746-4bab-907b-cf7930434a6d-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:07.093390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.093306    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"]
Sep 16 12:17:07.093636 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.093525    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy" containerID="cri-o://24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86" gracePeriod=30
Sep 16 12:17:07.111917 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.111885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:07.321871 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.321832    2577 generic.go:356] "Generic (PLEG): container finished" podID="81701c23-2746-4bab-907b-cf7930434a6d" containerID="c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60" exitCode=0
Sep 16 12:17:07.322098 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.321886    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"
Sep 16 12:17:07.322098 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.321911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" event={"ID":"81701c23-2746-4bab-907b-cf7930434a6d","Type":"ContainerDied","Data":"c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60"}
Sep 16 12:17:07.322098 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.321990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn" event={"ID":"81701c23-2746-4bab-907b-cf7930434a6d","Type":"ContainerDied","Data":"cb44ccf5869af3c6a9f3ad907351e6bd027bc932a0ac69863f3a806eb91b4b74"}
Sep 16 12:17:07.322098 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.322014    2577 scope.go:117] "RemoveContainer" containerID="c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60"
Sep 16 12:17:07.330881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.330861    2577 scope.go:117] "RemoveContainer" containerID="f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30"
Sep 16 12:17:07.340352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.340330    2577 scope.go:117] "RemoveContainer" containerID="c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60"
Sep 16 12:17:07.340743 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:07.340715    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60\": container with ID starting with c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60 not found: ID does not exist" containerID="c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60"
Sep 16 12:17:07.340822 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.340756    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60"} err="failed to get container status \"c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60\": rpc error: code = NotFound desc = could not find container \"c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60\": container with ID starting with c6c6447705c3fa854ceab1b5e6d84f5f277748c6343d34a9c5e21627a8457c60 not found: ID does not exist"
Sep 16 12:17:07.340822 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.340781    2577 scope.go:117] "RemoveContainer" containerID="f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30"
Sep 16 12:17:07.340904 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.340872    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"]
Sep 16 12:17:07.341083 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:07.341068    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30\": container with ID starting with f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30 not found: ID does not exist" containerID="f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30"
Sep 16 12:17:07.341118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.341088    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30"} err="failed to get container status \"f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30\": rpc error: code = NotFound desc = could not find container \"f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30\": container with ID starting with f09672d928d4409835c5a93ff2f7788713bdc7556dcd48c26705b9e0ea3fff30 not found: ID does not exist"
Sep 16 12:17:07.347376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.347324    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a1d14a87-openshift-default-8984bb744-mcnnn"]
Sep 16 12:17:07.630815 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.630781    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:07.938753 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.938669    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"]
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939020    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939031    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939041    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939047    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939057    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2565fe4e-20f5-4089-aa15-818b57d61d71" containerName="payload-pre-processing"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939064    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2565fe4e-20f5-4089-aa15-818b57d61d71" containerName="payload-pre-processing"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939080    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c557fe-04cc-4bde-ac84-62bfe1ea249f" containerName="maas-api"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939085    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c557fe-04cc-4bde-ac84-62bfe1ea249f" containerName="maas-api"
Sep 16 12:17:07.939172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939172    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c557fe-04cc-4bde-ac84-62bfe1ea249f" containerName="maas-api"
Sep 16 12:17:07.939682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939187    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2565fe4e-20f5-4089-aa15-818b57d61d71" containerName="payload-pre-processing"
Sep 16 12:17:07.939682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939199    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy"
Sep 16 12:17:07.939682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.939211    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81701c23-2746-4bab-907b-cf7930434a6d" containerName="istio-proxy"
Sep 16 12:17:07.944077 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.944035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:07.946720 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.946698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7810a5-openshift-default-dockercfg-r9tql\""
Sep 16 12:17:07.957681 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.957654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"]
Sep 16 12:17:07.973544 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:07.973517    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:08.085151 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istiod-ca-cert\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085329 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-socket\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085329 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-envoy\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-data\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjdl\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-kube-api-access-kpjdl\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-certs\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-podinfo\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085715 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-token\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.085715 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.085554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-credential-socket\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.142415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.142379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:08.186410 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-envoy\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.186620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-data\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.186620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjdl\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-kube-api-access-kpjdl\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.186620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-certs\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.186620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-podinfo\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.186876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-token\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.186876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-credential-socket\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.186876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istiod-ca-cert\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.187052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-socket\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.187052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-certs\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.187052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.186953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-data\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.187227 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.187150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-credential-socket\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.187227 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.187221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-socket\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.187492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.187473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istiod-ca-cert\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.189258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.189202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-envoy\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.189383 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.189350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-podinfo\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.194604 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.194561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-token\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.194765 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.194753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjdl\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-kube-api-access-kpjdl\") pod \"e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") " pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.257033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.256996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:08.530755 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.530717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"]
Sep 16 12:17:08.531158 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:17:08.531132    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6f484c_816a_46ab_bed8_74d3b50d2cc9.slice/crio-1c02716f41b3bd0d900f781b6a6c659b08a80b8769a56d6d37afc4e18a2bf195 WatchSource:0}: Error finding container 1c02716f41b3bd0d900f781b6a6c659b08a80b8769a56d6d37afc4e18a2bf195: Status 404 returned error can't find the container with id 1c02716f41b3bd0d900f781b6a6c659b08a80b8769a56d6d37afc4e18a2bf195
Sep 16 12:17:08.533374 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.533346    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:17:08.533449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.533407    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:17:08.533449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:08.533437    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:17:09.112455 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.112418    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81701c23-2746-4bab-907b-cf7930434a6d" path="/var/lib/kubelet/pods/81701c23-2746-4bab-907b-cf7930434a6d/volumes"
Sep 16 12:17:09.339097 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.339059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" event={"ID":"7f6f484c-816a-46ab-bed8-74d3b50d2cc9","Type":"ContainerStarted","Data":"4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee"}
Sep 16 12:17:09.339275 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.339125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" event={"ID":"7f6f484c-816a-46ab-bed8-74d3b50d2cc9","Type":"ContainerStarted","Data":"1c02716f41b3bd0d900f781b6a6c659b08a80b8769a56d6d37afc4e18a2bf195"}
Sep 16 12:17:09.360203 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.360144    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podStartSLOduration=2.360123065 podStartE2EDuration="2.360123065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:09.358633137 +0000 UTC m=+916.770650032" watchObservedRunningTime="2026-09-16 12:17:09.360123065 +0000 UTC m=+916.772139963"
Sep 16 12:17:09.576650 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.576611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:09.677009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.676979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:09.776419 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.776381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:09.827631 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.827532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:09.889599 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:09.889542    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:10.258192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:10.258152    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:10.263538 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:10.263508    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:10.345330 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:10.345295    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:10.346273 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:10.346255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:10.404393 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:10.404325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:10.699484 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:10.699451    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:11.302366 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:11.302332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:11.461861 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:11.461823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:11.558081 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:11.557971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:12.140875 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.140838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:12.344661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.344633    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:17:12.355881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.355806    2577 generic.go:356] "Generic (PLEG): container finished" podID="619093be-c45c-48cd-a541-f6eb6df4323c" containerID="24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86" exitCode=0
Sep 16 12:17:12.355881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.355854    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"
Sep 16 12:17:12.355881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.355853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" event={"ID":"619093be-c45c-48cd-a541-f6eb6df4323c","Type":"ContainerDied","Data":"24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86"}
Sep 16 12:17:12.356063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.355941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4" event={"ID":"619093be-c45c-48cd-a541-f6eb6df4323c","Type":"ContainerDied","Data":"cfbcbb450a15c46ad2b8ae12c055c6778ebf2ff64b240a4bbbe1b2a0300f3d8f"}
Sep 16 12:17:12.356063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.355966    2577 scope.go:117] "RemoveContainer" containerID="24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86"
Sep 16 12:17:12.367147 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.366360    2577 scope.go:117] "RemoveContainer" containerID="6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4"
Sep 16 12:17:12.378183 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.378162    2577 scope.go:117] "RemoveContainer" containerID="24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86"
Sep 16 12:17:12.378505 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:12.378481    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86\": container with ID starting with 24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86 not found: ID does not exist" containerID="24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86"
Sep 16 12:17:12.378649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.378511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86"} err="failed to get container status \"24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86\": rpc error: code = NotFound desc = could not find container \"24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86\": container with ID starting with 24120dc480aa8e910d2e49f92727125c29f810d1e9c79130fe93b98f28cc9a86 not found: ID does not exist"
Sep 16 12:17:12.378649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.378529    2577 scope.go:117] "RemoveContainer" containerID="6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4"
Sep 16 12:17:12.378841 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:12.378821    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4\": container with ID starting with 6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4 not found: ID does not exist" containerID="6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4"
Sep 16 12:17:12.378911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.378845    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4"} err="failed to get container status \"6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4\": rpc error: code = NotFound desc = could not find container \"6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4\": container with ID starting with 6358ab612f05a91c405d8250a1c12a2db0b50a77adbb36f56804182c101689e4 not found: ID does not exist"
Sep 16 12:17:12.527744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527701    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619093be-c45c-48cd-a541-f6eb6df4323c-istiod-ca-cert\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.527928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-envoy\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.527928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-data\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.527928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nn7c\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-kube-api-access-5nn7c\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.527928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-istio-token\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.527928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-certs\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.528198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-credential-socket\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.528198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.527970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-socket\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.528198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.528025    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619093be-c45c-48cd-a541-f6eb6df4323c-istio-podinfo\") pod \"619093be-c45c-48cd-a541-f6eb6df4323c\" (UID: \"619093be-c45c-48cd-a541-f6eb6df4323c\") "
Sep 16 12:17:12.528198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.528184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:12.528415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.528390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:12.528415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.528394    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.528527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.528486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-data" (OuterVolumeSpecName: "istio-data") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:12.528634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.528610    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:12.529063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.529034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619093be-c45c-48cd-a541-f6eb6df4323c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:17:12.530329 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.530297    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:17:12.530433 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.530363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-kube-api-access-5nn7c" (OuterVolumeSpecName: "kube-api-access-5nn7c") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "kube-api-access-5nn7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:17:12.530855 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.530838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/619093be-c45c-48cd-a541-f6eb6df4323c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:17:12.530947 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.530872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-istio-token" (OuterVolumeSpecName: "istio-token") pod "619093be-c45c-48cd-a541-f6eb6df4323c" (UID: "619093be-c45c-48cd-a541-f6eb6df4323c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:17:12.629334 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629298    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.629489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629336    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nn7c\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-kube-api-access-5nn7c\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.629489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629348    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619093be-c45c-48cd-a541-f6eb6df4323c-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.629489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629356    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.629489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629365    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.629489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629373    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619093be-c45c-48cd-a541-f6eb6df4323c-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.629489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629381    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619093be-c45c-48cd-a541-f6eb6df4323c-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.629489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.629389    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619093be-c45c-48cd-a541-f6eb6df4323c-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:17:12.679796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.679525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"]
Sep 16 12:17:12.681785 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.681759    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-dc3e2c-openshift-default-68565ccffd-p82j4"]
Sep 16 12:17:12.860154 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:12.860111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:13.112255 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:13.112171    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" path="/var/lib/kubelet/pods/619093be-c45c-48cd-a541-f6eb6df4323c/volumes"
Sep 16 12:17:13.560340 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:13.560294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:13.983428 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:13.983391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:17:14.144732 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.144699    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"]
Sep 16 12:17:14.145423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.145398    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy"
Sep 16 12:17:14.145423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.145424    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy"
Sep 16 12:17:14.145563 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.145537    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy"
Sep 16 12:17:14.150525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.150503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.153729 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.153705    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:17:14.155057 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.155033    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7fbn\""
Sep 16 12:17:14.155288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.155149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:17:14.155288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.155149    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:17:14.155529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.155510    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-7810a5-serving-cert\""
Sep 16 12:17:14.158214 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.158155    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"]
Sep 16 12:17:14.173118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.173093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:14.259386 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.259209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:14.346880 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.346847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-metrics-certs\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.347033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.346930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5d8\" (UniqueName: \"kubernetes.io/projected/900fea22-b796-461e-9942-bda574ba70a6-kube-api-access-gk5d8\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.347033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.346957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.347033 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.346975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900fea22-b796-461e-9942-bda574ba70a6-openshift-service-ca\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.448261 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.448222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5d8\" (UniqueName: \"kubernetes.io/projected/900fea22-b796-461e-9942-bda574ba70a6-kube-api-access-gk5d8\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.448441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.448273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.448441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.448295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900fea22-b796-461e-9942-bda574ba70a6-openshift-service-ca\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.448441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.448338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-metrics-certs\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.448441 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:14.448430    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-serving-cert: secret "maas-api-e2e-explicit-7810a5-serving-cert" not found
Sep 16 12:17:14.448744 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:14.448512    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls podName:900fea22-b796-461e-9942-bda574ba70a6 nodeName:}" failed. No retries permitted until 2026-09-16 12:17:14.948491039 +0000 UTC m=+922.360507927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls") pod "maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" (UID: "900fea22-b796-461e-9942-bda574ba70a6") : secret "maas-api-e2e-explicit-7810a5-serving-cert" not found
Sep 16 12:17:14.449091 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.449068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900fea22-b796-461e-9942-bda574ba70a6-openshift-service-ca\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.450678 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.450659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-metrics-certs\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.456080 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.456054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5d8\" (UniqueName: \"kubernetes.io/projected/900fea22-b796-461e-9942-bda574ba70a6-kube-api-access-gk5d8\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.593836 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.593728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:14.682097 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.682063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:14.686273 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.686243    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"]
Sep 16 12:17:14.686639 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.686625    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy"
Sep 16 12:17:14.686737 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.686642    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy"
Sep 16 12:17:14.686737 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.686714    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="619093be-c45c-48cd-a541-f6eb6df4323c" containerName="istio-proxy"
Sep 16 12:17:14.689822 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.689806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:14.692435 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.692410    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7810a5-dockercfg-wdmth\""
Sep 16 12:17:14.692601 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.692414    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7810a5\""
Sep 16 12:17:14.701795 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.701770    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"]
Sep 16 12:17:14.852942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.852836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d285a2d7-722b-47ef-965d-67aaa20c18ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:14.853132 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.852959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zf7r\" (UniqueName: \"kubernetes.io/projected/d285a2d7-722b-47ef-965d-67aaa20c18ef-kube-api-access-8zf7r\") pod \"payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:14.954445 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.954386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d285a2d7-722b-47ef-965d-67aaa20c18ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:14.954651 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.954483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:14.954651 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.954531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zf7r\" (UniqueName: \"kubernetes.io/projected/d285a2d7-722b-47ef-965d-67aaa20c18ef-kube-api-access-8zf7r\") pod \"payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:14.954746 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:14.954668    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-serving-cert: secret "maas-api-e2e-explicit-7810a5-serving-cert" not found
Sep 16 12:17:14.954789 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:17:14.954781    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls podName:900fea22-b796-461e-9942-bda574ba70a6 nodeName:}" failed. No retries permitted until 2026-09-16 12:17:15.954758788 +0000 UTC m=+923.366775669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls") pod "maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" (UID: "900fea22-b796-461e-9942-bda574ba70a6") : secret "maas-api-e2e-explicit-7810a5-serving-cert" not found
Sep 16 12:17:14.955215 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.955198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d285a2d7-722b-47ef-965d-67aaa20c18ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:14.962094 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:14.962070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zf7r\" (UniqueName: \"kubernetes.io/projected/d285a2d7-722b-47ef-965d-67aaa20c18ef-kube-api-access-8zf7r\") pod \"payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:15.002220 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.002183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:15.275454 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.275421    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"]
Sep 16 12:17:15.276517 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:17:15.276493    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd285a2d7_722b_47ef_965d_67aaa20c18ef.slice/crio-acbe76775934aafb9ac5f893dd09b95cf73d2b6472247bb5b5964be7ddeae95c WatchSource:0}: Error finding container acbe76775934aafb9ac5f893dd09b95cf73d2b6472247bb5b5964be7ddeae95c: Status 404 returned error can't find the container with id acbe76775934aafb9ac5f893dd09b95cf73d2b6472247bb5b5964be7ddeae95c
Sep 16 12:17:15.292441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.292415    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:15.374968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.374927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7" event={"ID":"d285a2d7-722b-47ef-965d-67aaa20c18ef","Type":"ContainerStarted","Data":"a166d2708cbf0753ef1418b07694d6dc17f560f9821707287d875725c6caa0a0"}
Sep 16 12:17:15.375128 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.374997    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:15.375128 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.375027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7" event={"ID":"d285a2d7-722b-47ef-965d-67aaa20c18ef","Type":"ContainerStarted","Data":"acbe76775934aafb9ac5f893dd09b95cf73d2b6472247bb5b5964be7ddeae95c"}
Sep 16 12:17:15.394691 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.394636    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7" podStartSLOduration=1.394615997 podStartE2EDuration="1.394615997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:15.393821435 +0000 UTC m=+922.805838332" watchObservedRunningTime="2026-09-16 12:17:15.394615997 +0000 UTC m=+922.806632893"
Sep 16 12:17:15.962130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.962088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:15.964437 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:15.964410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls\") pod \"maas-api-e2e-explicit-7810a5-74c6c68c96-q249f\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:16.264060 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:16.263957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:16.535502 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:16.535476    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"]
Sep 16 12:17:16.537958 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:17:16.537926    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900fea22_b796_461e_9942_bda574ba70a6.slice/crio-c20121ff680b9bea68ee4371ae4225e0d57b52223dead67e82b21ca4483d4543 WatchSource:0}: Error finding container c20121ff680b9bea68ee4371ae4225e0d57b52223dead67e82b21ca4483d4543: Status 404 returned error can't find the container with id c20121ff680b9bea68ee4371ae4225e0d57b52223dead67e82b21ca4483d4543
Sep 16 12:17:17.387238 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:17.387198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" event={"ID":"900fea22-b796-461e-9942-bda574ba70a6","Type":"ContainerStarted","Data":"66e32923d046882d90c077f4996be72b992e9c42caf1997a8f9276aa2a1ad60e"}
Sep 16 12:17:17.387664 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:17.387266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" event={"ID":"900fea22-b796-461e-9942-bda574ba70a6","Type":"ContainerStarted","Data":"c20121ff680b9bea68ee4371ae4225e0d57b52223dead67e82b21ca4483d4543"}
Sep 16 12:17:17.387664 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:17.387411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:17.404714 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:17.404667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" podStartSLOduration=3.001183463 podStartE2EDuration="3.404652528s" totalImagesPullingTime="403.469065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:17.403293739 +0000 UTC m=+924.815310636" watchObservedRunningTime="2026-09-16 12:17:17.404652528 +0000 UTC m=+924.816669423"
Sep 16 12:17:23.397948 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:23.397916    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:17:25.846858 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.846825    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"]
Sep 16 12:17:25.850672 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.850649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:25.853397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.853376    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"kube-root-ca.crt\""
Sep 16 12:17:25.854636 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.854607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"openshift-service-ca.crt\""
Sep 16 12:17:25.854764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.854608    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"default-dockercfg-szmz8\""
Sep 16 12:17:25.854764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.854612    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"premium-w0-ff057a-kserve-self-signed-certs\""
Sep 16 12:17:25.860153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.860131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"]
Sep 16 12:17:25.879238 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.879210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:25.945634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.945593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-home\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:25.945783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.945647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-tmp-dir\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:25.945783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.945695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447hh\" (UniqueName: \"kubernetes.io/projected/069b2328-0a36-4957-8529-cf544bc62bbf-kube-api-access-447hh\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:25.945783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.945747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-model-cache\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:25.945783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.945771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-dshm\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:25.945939 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:25.945795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/069b2328-0a36-4957-8529-cf544bc62bbf-tls-certs\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.046786 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.046751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-model-cache\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.046975 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.046797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-dshm\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.046975 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.046828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/069b2328-0a36-4957-8529-cf544bc62bbf-tls-certs\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.046975 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.046890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-home\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.046975 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.046922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-tmp-dir\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.046975 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.046966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447hh\" (UniqueName: \"kubernetes.io/projected/069b2328-0a36-4957-8529-cf544bc62bbf-kube-api-access-447hh\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.047238 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.047221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-model-cache\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.047360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.047313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-home\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.047360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.047353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-tmp-dir\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.049388 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.049360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-dshm\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.049542 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.049524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/069b2328-0a36-4957-8529-cf544bc62bbf-tls-certs\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.071863 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.071831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-447hh\" (UniqueName: \"kubernetes.io/projected/069b2328-0a36-4957-8529-cf544bc62bbf-kube-api-access-447hh\") pod \"premium-w0-ff057a-kserve-dbc64869c-6759c\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") " pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.161754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.161714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:26.383143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.383114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:17:26.431887 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:26.431843    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"]
Sep 16 12:17:26.434351 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:17:26.434317    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod069b2328_0a36_4957_8529_cf544bc62bbf.slice/crio-024c3fa7795b1cebcdb2e8d7bccb736685ca334cd175f998f210dfba7eb36128 WatchSource:0}: Error finding container 024c3fa7795b1cebcdb2e8d7bccb736685ca334cd175f998f210dfba7eb36128: Status 404 returned error can't find the container with id 024c3fa7795b1cebcdb2e8d7bccb736685ca334cd175f998f210dfba7eb36128
Sep 16 12:17:27.446922 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:27.446867    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c" event={"ID":"069b2328-0a36-4957-8529-cf544bc62bbf","Type":"ContainerStarted","Data":"dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b"}
Sep 16 12:17:27.446922 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:27.446933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c" event={"ID":"069b2328-0a36-4957-8529-cf544bc62bbf","Type":"ContainerStarted","Data":"024c3fa7795b1cebcdb2e8d7bccb736685ca334cd175f998f210dfba7eb36128"}
Sep 16 12:17:27.465744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:27.465685    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c" podStartSLOduration=2.465669815 podStartE2EDuration="2.465669815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:27.463844433 +0000 UTC m=+934.875861323" watchObservedRunningTime="2026-09-16 12:17:27.465669815 +0000 UTC m=+934.877686713"
Sep 16 12:17:35.185038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:35.184996    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:17:35.493365 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:35.493267    2577 generic.go:356] "Generic (PLEG): container finished" podID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerID="4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee" exitCode=0
Sep 16 12:17:35.493365 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:35.493336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" event={"ID":"7f6f484c-816a-46ab-bed8-74d3b50d2cc9","Type":"ContainerDied","Data":"4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee"}
Sep 16 12:17:35.493751 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:35.493735    2577 scope.go:117] "RemoveContainer" containerID="4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee"
Sep 16 12:17:35.494903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:35.494861    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:17:35.494987 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:35.494947    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:17:35.494987 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:35.494980    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:17:36.162635 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:36.162604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:36.162825 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:36.162671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:36.177052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:36.177024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:36.463397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:36.463297    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:36.500780 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:36.500737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" event={"ID":"7f6f484c-816a-46ab-bed8-74d3b50d2cc9","Type":"ContainerStarted","Data":"cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c"}
Sep 16 12:17:36.511940 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:36.511901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:17:36.784146 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:36.784056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:37.257443 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:37.257401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:37.258714 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:37.258680    2577 patch_prober.go:28] interesting pod/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:17:37.258849 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:37.258737    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 12:17:37.671820 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:37.671785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:38.256830 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:38.256787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:38.257821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:38.257798    2577 patch_prober.go:28] interesting pod/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:17:38.257878 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:38.257854    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 12:17:38.846208 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:38.846166    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:39.257625 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:39.257586    2577 patch_prober.go:28] interesting pod/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:17:39.257896 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:39.257646    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 12:17:39.579765 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:39.579671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:40.169874 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:40.169837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:40.257941 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:40.257896    2577 patch_prober.go:28] interesting pod/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:17:40.258117 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:40.257992    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 12:17:40.346122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:40.346072    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:41.258286 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:41.258244    2577 patch_prober.go:28] interesting pod/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:17:41.258726 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:41.258308    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 12:17:41.691180 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:41.691151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:42.262094 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:42.262048    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:42.263128 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:42.263102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:17:42.496505 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:42.496474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:42.555817 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:42.555709    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:42.645280 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:42.645241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:42.742736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:42.742699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:49.344497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:49.344464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:59.080042 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:59.080004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:17:59.990629 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:17:59.990590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:18:04.268197 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:04.268159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:04.700655 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:04.700619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:04.765708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:04.765672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:04.913421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:04.913383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:05.017651 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:05.017537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:15.895426 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:15.895343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:16.985067 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:16.985032    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:17.581721 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:17.581683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:18:19.851246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:19.851214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:22.587071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:22.587032    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:22.732218 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:22.732119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:22.822634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:22.822561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:22.922718 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:22.922687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:23.691953 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:23.691916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:23.810286 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:23.810253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:23.939663 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:23.939631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:25.822795 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:25.822759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:28.667808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:28.667771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:31.484691 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:31.484656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:31.584529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:31.584490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:31.917338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:31.917303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:32.339422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:32.339340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:34.590313 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:34.590281    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:18:38.020273 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:38.020236    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:38.121019 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:38.120987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:38.319062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:38.318968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:38.663447 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:38.663410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:38.843771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:38.843738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:44.186032 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:44.185997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:44.909529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:44.909491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:45.014136 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:45.014084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:45.115758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:45.115712    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:45.774359 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:45.774320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:46.395746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:46.395697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:47.000886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:47.000852    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:47.766720 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:47.766652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:48.378124 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.378086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:48.614910 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.614874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:48.715461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.715383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:48.818086 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.818049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:48.894528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.894492    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:18:48.900871 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.900836    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"]
Sep 16 12:18:48.905018 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.904988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:48.907891 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.907856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f916d574-dockercfg-4zbhm\""
Sep 16 12:18:48.908020 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.907909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f916d574\""
Sep 16 12:18:48.915061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.915038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"]
Sep 16 12:18:48.935931 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.935902    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"]
Sep 16 12:18:48.938389 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.938365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:48.947588 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.947545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"]
Sep 16 12:18:48.982495 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.982346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:48.982849 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:48.982814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjn9h\" (UniqueName: \"kubernetes.io/projected/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-kube-api-access-kjn9h\") pod \"payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:49.084481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.084433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:49.084481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.084483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjn9h\" (UniqueName: \"kubernetes.io/projected/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-kube-api-access-kjn9h\") pod \"payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:49.084774 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.084513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9p4r\" (UniqueName: \"kubernetes.io/projected/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-kube-api-access-q9p4r\") pod \"payload-processing-e2e-mt-f916d574-579c745c69-6nmbx\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") " pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.084774 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.084539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f916d574-579c745c69-6nmbx\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") " pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.085191 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.085167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:49.092819 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.092797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjn9h\" (UniqueName: \"kubernetes.io/projected/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-kube-api-access-kjn9h\") pod \"payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:49.186073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.186025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9p4r\" (UniqueName: \"kubernetes.io/projected/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-kube-api-access-q9p4r\") pod \"payload-processing-e2e-mt-f916d574-579c745c69-6nmbx\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") " pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.186288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.186084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f916d574-579c745c69-6nmbx\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") " pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.186886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.186859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f916d574-579c745c69-6nmbx\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") " pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.193613 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.193590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9p4r\" (UniqueName: \"kubernetes.io/projected/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-kube-api-access-q9p4r\") pod \"payload-processing-e2e-mt-f916d574-579c745c69-6nmbx\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") " pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.216505 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.216475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:49.248671 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.248604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.497891 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.497865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"]
Sep 16 12:18:49.499117 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:18:49.499067    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a5e9de_fe6e_4b7e_896e_26ec9a9a64e9.slice/crio-42ab2c3422b6aad3fee04f40f1c805d699d66b0009f19674dfc5359b7960c506 WatchSource:0}: Error finding container 42ab2c3422b6aad3fee04f40f1c805d699d66b0009f19674dfc5359b7960c506: Status 404 returned error can't find the container with id 42ab2c3422b6aad3fee04f40f1c805d699d66b0009f19674dfc5359b7960c506
Sep 16 12:18:49.528837 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.528809    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"]
Sep 16 12:18:49.532892 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:18:49.532859    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbbef58_2f00_4c9a_a24e_fd6b547a97c5.slice/crio-465ab856b5d6a404886406f058f14f8e2d96068a08306fca8095cd765bdc1fe0 WatchSource:0}: Error finding container 465ab856b5d6a404886406f058f14f8e2d96068a08306fca8095cd765bdc1fe0: Status 404 returned error can't find the container with id 465ab856b5d6a404886406f058f14f8e2d96068a08306fca8095cd765bdc1fe0
Sep 16 12:18:49.892008 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.891972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5" event={"ID":"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9","Type":"ContainerStarted","Data":"35b198a65fe6714f5fcba3e02008a0231fd33d7390657ed5031b534ef097badb"}
Sep 16 12:18:49.892229 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.892036    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:18:49.892229 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.892060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5" event={"ID":"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9","Type":"ContainerStarted","Data":"42ab2c3422b6aad3fee04f40f1c805d699d66b0009f19674dfc5359b7960c506"}
Sep 16 12:18:49.893677 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.893649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx" event={"ID":"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5","Type":"ContainerStarted","Data":"b8196aaa896b2c8f66589a36d9873e3e7e978a2f1992d5591a4991d735246bce"}
Sep 16 12:18:49.893803 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.893698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx" event={"ID":"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5","Type":"ContainerStarted","Data":"465ab856b5d6a404886406f058f14f8e2d96068a08306fca8095cd765bdc1fe0"}
Sep 16 12:18:49.893803 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.893736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:18:49.908060 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.908006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5" podStartSLOduration=1.907989801 podStartE2EDuration="1.907989801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:49.906011873 +0000 UTC m=+1017.318028772" watchObservedRunningTime="2026-09-16 12:18:49.907989801 +0000 UTC m=+1017.320006697"
Sep 16 12:18:49.922753 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:18:49.922705    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx" podStartSLOduration=1.922688792 podStartE2EDuration="1.922688792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:18:49.92098379 +0000 UTC m=+1017.333000687" watchObservedRunningTime="2026-09-16 12:18:49.922688792 +0000 UTC m=+1017.334705687"
Sep 16 12:19:00.679356 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:00.679325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:00.901782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:00.901740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:19:00.902041 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:00.902018    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:19:03.456743 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:03.456705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:03.764146 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:03.764061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:04.461866 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:04.461804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:05.075403 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:05.075372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:05.878016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:05.877976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:06.484771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:06.484739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:07.271504 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:07.271472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:07.778795 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:07.778756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:08.494871 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:08.494836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:08.794652 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:08.794536    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:19:09.379803 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:09.379766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:10.035742 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:10.035697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:10.135172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:10.135137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:10.758661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:10.758611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:10.898452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:10.898413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:11.050747 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:11.050646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:11.749611 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:11.749579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:12.596921 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:12.596891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:13.264386 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:13.264351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:13.796801 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:13.796766    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"]
Sep 16 12:19:13.797283 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:13.796967    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" podUID="900fea22-b796-461e-9942-bda574ba70a6" containerName="maas-api" containerID="cri-o://66e32923d046882d90c077f4996be72b992e9c42caf1997a8f9276aa2a1ad60e" gracePeriod=30
Sep 16 12:19:13.827744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:13.827697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:13.869461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:13.869426    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"]
Sep 16 12:19:13.869848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:13.869706    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7" podUID="d285a2d7-722b-47ef-965d-67aaa20c18ef" containerName="payload-pre-processing" containerID="cri-o://a166d2708cbf0753ef1418b07694d6dc17f560f9821707287d875725c6caa0a0" gracePeriod=30
Sep 16 12:19:13.927021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:13.926986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:14.024980 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.024946    2577 generic.go:356] "Generic (PLEG): container finished" podID="d285a2d7-722b-47ef-965d-67aaa20c18ef" containerID="a166d2708cbf0753ef1418b07694d6dc17f560f9821707287d875725c6caa0a0" exitCode=0
Sep 16 12:19:14.025156 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.025075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7" event={"ID":"d285a2d7-722b-47ef-965d-67aaa20c18ef","Type":"ContainerDied","Data":"a166d2708cbf0753ef1418b07694d6dc17f560f9821707287d875725c6caa0a0"}
Sep 16 12:19:14.026714 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.026685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:14.026962 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.026937    2577 generic.go:356] "Generic (PLEG): container finished" podID="900fea22-b796-461e-9942-bda574ba70a6" containerID="66e32923d046882d90c077f4996be72b992e9c42caf1997a8f9276aa2a1ad60e" exitCode=0
Sep 16 12:19:14.027070 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.027031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" event={"ID":"900fea22-b796-461e-9942-bda574ba70a6","Type":"ContainerDied","Data":"66e32923d046882d90c077f4996be72b992e9c42caf1997a8f9276aa2a1ad60e"}
Sep 16 12:19:14.090831 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.090782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:14.122545 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.122135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:19:14.162122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.162089    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:19:14.199931 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.199887    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900fea22-b796-461e-9942-bda574ba70a6-openshift-service-ca\") pod \"900fea22-b796-461e-9942-bda574ba70a6\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") "
Sep 16 12:19:14.200125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.199951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-metrics-certs\") pod \"900fea22-b796-461e-9942-bda574ba70a6\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") "
Sep 16 12:19:14.200125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.200007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls\") pod \"900fea22-b796-461e-9942-bda574ba70a6\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") "
Sep 16 12:19:14.200125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.200030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zf7r\" (UniqueName: \"kubernetes.io/projected/d285a2d7-722b-47ef-965d-67aaa20c18ef-kube-api-access-8zf7r\") pod \"d285a2d7-722b-47ef-965d-67aaa20c18ef\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") "
Sep 16 12:19:14.200125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.200066    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk5d8\" (UniqueName: \"kubernetes.io/projected/900fea22-b796-461e-9942-bda574ba70a6-kube-api-access-gk5d8\") pod \"900fea22-b796-461e-9942-bda574ba70a6\" (UID: \"900fea22-b796-461e-9942-bda574ba70a6\") "
Sep 16 12:19:14.200125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.200101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d285a2d7-722b-47ef-965d-67aaa20c18ef-plugins-config-volume\") pod \"d285a2d7-722b-47ef-965d-67aaa20c18ef\" (UID: \"d285a2d7-722b-47ef-965d-67aaa20c18ef\") "
Sep 16 12:19:14.200387 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.200351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900fea22-b796-461e-9942-bda574ba70a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "900fea22-b796-461e-9942-bda574ba70a6" (UID: "900fea22-b796-461e-9942-bda574ba70a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:19:14.200738 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.200711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d285a2d7-722b-47ef-965d-67aaa20c18ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d285a2d7-722b-47ef-965d-67aaa20c18ef" (UID: "d285a2d7-722b-47ef-965d-67aaa20c18ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:19:14.202423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.202398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900fea22-b796-461e-9942-bda574ba70a6-kube-api-access-gk5d8" (OuterVolumeSpecName: "kube-api-access-gk5d8") pod "900fea22-b796-461e-9942-bda574ba70a6" (UID: "900fea22-b796-461e-9942-bda574ba70a6"). InnerVolumeSpecName "kube-api-access-gk5d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:14.202531 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.202431    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d285a2d7-722b-47ef-965d-67aaa20c18ef-kube-api-access-8zf7r" (OuterVolumeSpecName: "kube-api-access-8zf7r") pod "d285a2d7-722b-47ef-965d-67aaa20c18ef" (UID: "d285a2d7-722b-47ef-965d-67aaa20c18ef"). InnerVolumeSpecName "kube-api-access-8zf7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:14.202706 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.202685    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "900fea22-b796-461e-9942-bda574ba70a6" (UID: "900fea22-b796-461e-9942-bda574ba70a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:14.202771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.202700    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "900fea22-b796-461e-9942-bda574ba70a6" (UID: "900fea22-b796-461e-9942-bda574ba70a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:19:14.300981 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.300940    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5d8\" (UniqueName: \"kubernetes.io/projected/900fea22-b796-461e-9942-bda574ba70a6-kube-api-access-gk5d8\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:14.300981 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.300975    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d285a2d7-722b-47ef-965d-67aaa20c18ef-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:14.300981 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.300986    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/900fea22-b796-461e-9942-bda574ba70a6-openshift-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:14.301222 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.300996    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-metrics-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:14.301222 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.301006    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/900fea22-b796-461e-9942-bda574ba70a6-maas-api-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:14.301222 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.301014    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zf7r\" (UniqueName: \"kubernetes.io/projected/d285a2d7-722b-47ef-965d-67aaa20c18ef-kube-api-access-8zf7r\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:14.436385 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.436351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:14.594710 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:14.594669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:15.033783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.033751    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"
Sep 16 12:19:15.034393 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.033788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f" event={"ID":"900fea22-b796-461e-9942-bda574ba70a6","Type":"ContainerDied","Data":"c20121ff680b9bea68ee4371ae4225e0d57b52223dead67e82b21ca4483d4543"}
Sep 16 12:19:15.034393 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.033864    2577 scope.go:117] "RemoveContainer" containerID="66e32923d046882d90c077f4996be72b992e9c42caf1997a8f9276aa2a1ad60e"
Sep 16 12:19:15.035497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.035437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7" event={"ID":"d285a2d7-722b-47ef-965d-67aaa20c18ef","Type":"ContainerDied","Data":"acbe76775934aafb9ac5f893dd09b95cf73d2b6472247bb5b5964be7ddeae95c"}
Sep 16 12:19:15.035497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.035481    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"
Sep 16 12:19:15.044075 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.044057    2577 scope.go:117] "RemoveContainer" containerID="a166d2708cbf0753ef1418b07694d6dc17f560f9821707287d875725c6caa0a0"
Sep 16 12:19:15.060113 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.060085    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"]
Sep 16 12:19:15.063717 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.063660    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7810a5-74c6c68c96-q249f"]
Sep 16 12:19:15.074322 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.074288    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"]
Sep 16 12:19:15.078467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.078435    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7810a5-548f95d998-xpsz7"]
Sep 16 12:19:15.111769 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.111720    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="900fea22-b796-461e-9942-bda574ba70a6" path="/var/lib/kubelet/pods/900fea22-b796-461e-9942-bda574ba70a6/volumes"
Sep 16 12:19:15.112219 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.112197    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d285a2d7-722b-47ef-965d-67aaa20c18ef" path="/var/lib/kubelet/pods/d285a2d7-722b-47ef-965d-67aaa20c18ef/volumes"
Sep 16 12:19:15.281547 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:15.281508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:16.185814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:16.185776    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:16.287402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:16.287370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:16.808214 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:16.808176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:17.570117 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:17.570073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:18.287262 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:18.287225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:20.045394 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:20.045359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:20.146169 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:20.146137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:21.979968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:21.979932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:22.079058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:22.079025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:25.288902 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:25.288867    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"]
Sep 16 12:19:25.289267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:25.289088    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy" containerID="cri-o://cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c" gracePeriod=30
Sep 16 12:19:25.304073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:25.304042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:27.265287 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:27.265242    2577 patch_prober.go:28] interesting pod/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:19:27.265677 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:27.265313    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:19:30.544918 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.544892    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:19:30.649837 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.649800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpjdl\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-kube-api-access-kpjdl\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650007 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.649857    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-token\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650007 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.649883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-envoy\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650007 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.649906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istiod-ca-cert\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650007 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.649925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-data\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650007 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.649997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-socket\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-podinfo\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-credential-socket\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-certs\") pod \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\" (UID: \"7f6f484c-816a-46ab-bed8-74d3b50d2cc9\") "
Sep 16 12:19:30.650554 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:30.650554 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650471    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:30.650554 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650522    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:30.650809 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:19:30.650809 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.650728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:30.652632 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.652607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:19:30.652632 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.652622    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-kube-api-access-kpjdl" (OuterVolumeSpecName: "kube-api-access-kpjdl") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "kube-api-access-kpjdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:30.652799 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.652647    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:19:30.652799 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.652702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f6f484c-816a-46ab-bed8-74d3b50d2cc9" (UID: "7f6f484c-816a-46ab-bed8-74d3b50d2cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:30.751401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751350    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751397    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751411    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751422    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751431    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751440    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751448    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjdl\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-kube-api-access-kpjdl\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751457    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:30.751728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:30.751465    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6f484c-816a-46ab-bed8-74d3b50d2cc9-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:31.123066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.123030    2577 generic.go:356] "Generic (PLEG): container finished" podID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerID="cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c" exitCode=0
Sep 16 12:19:31.123174 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.123080    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"
Sep 16 12:19:31.123174 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.123109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" event={"ID":"7f6f484c-816a-46ab-bed8-74d3b50d2cc9","Type":"ContainerDied","Data":"cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c"}
Sep 16 12:19:31.123174 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.123162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7" event={"ID":"7f6f484c-816a-46ab-bed8-74d3b50d2cc9","Type":"ContainerDied","Data":"1c02716f41b3bd0d900f781b6a6c659b08a80b8769a56d6d37afc4e18a2bf195"}
Sep 16 12:19:31.123297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.123184    2577 scope.go:117] "RemoveContainer" containerID="cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c"
Sep 16 12:19:31.132003 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.131982    2577 scope.go:117] "RemoveContainer" containerID="4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee"
Sep 16 12:19:31.140487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.140460    2577 scope.go:117] "RemoveContainer" containerID="cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c"
Sep 16 12:19:31.141018 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:19:31.140987    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c\": container with ID starting with cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c not found: ID does not exist" containerID="cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c"
Sep 16 12:19:31.141137 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.141032    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c"} err="failed to get container status \"cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c\": rpc error: code = NotFound desc = could not find container \"cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c\": container with ID starting with cdf049dc07bfebde04c557c15e0571963345dd3c8dd4e3f2c88bc677360ba16c not found: ID does not exist"
Sep 16 12:19:31.141137 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.141060    2577 scope.go:117] "RemoveContainer" containerID="4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee"
Sep 16 12:19:31.141481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.141456    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"]
Sep 16 12:19:31.141890 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:19:31.141753    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee\": container with ID starting with 4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee not found: ID does not exist" containerID="4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee"
Sep 16 12:19:31.141890 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.141789    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee"} err="failed to get container status \"4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee\": rpc error: code = NotFound desc = could not find container \"4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee\": container with ID starting with 4b3c569ec0e7c265c930243db51b9565ffd54d8112c9db0c0b68cbadcc14daee not found: ID does not exist"
Sep 16 12:19:31.143627 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:31.143609    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7810a5-openshift-default-79c6f9c4b4-pbtd7"]
Sep 16 12:19:33.111649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:33.111605    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" path="/var/lib/kubelet/pods/7f6f484c-816a-46ab-bed8-74d3b50d2cc9/volumes"
Sep 16 12:19:35.813079 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:35.813042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:37.263694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.263656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:37.345986 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.345953    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"]
Sep 16 12:19:37.346328 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346316    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy"
Sep 16 12:19:37.346379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346331    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy"
Sep 16 12:19:37.346379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346339    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d285a2d7-722b-47ef-965d-67aaa20c18ef" containerName="payload-pre-processing"
Sep 16 12:19:37.346379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346345    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d285a2d7-722b-47ef-965d-67aaa20c18ef" containerName="payload-pre-processing"
Sep 16 12:19:37.346379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346356    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="900fea22-b796-461e-9942-bda574ba70a6" containerName="maas-api"
Sep 16 12:19:37.346379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346363    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="900fea22-b796-461e-9942-bda574ba70a6" containerName="maas-api"
Sep 16 12:19:37.346379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346370    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy"
Sep 16 12:19:37.346379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346375    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy"
Sep 16 12:19:37.346649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346432    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="900fea22-b796-461e-9942-bda574ba70a6" containerName="maas-api"
Sep 16 12:19:37.346649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346441    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d285a2d7-722b-47ef-965d-67aaa20c18ef" containerName="payload-pre-processing"
Sep 16 12:19:37.346649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346448    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy"
Sep 16 12:19:37.346649 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.346455    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f6f484c-816a-46ab-bed8-74d3b50d2cc9" containerName="istio-proxy"
Sep 16 12:19:37.349026 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.349007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"
Sep 16 12:19:37.351856 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.351834    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:19:37.352005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.351832    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:19:37.356796 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.356775    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"]
Sep 16 12:19:37.511394 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.511358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c082cbec-6411-4eae-9a34-94ede034cf88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f916d574-hdv2g\" (UID: \"c082cbec-6411-4eae-9a34-94ede034cf88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"
Sep 16 12:19:37.612792 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.612706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c082cbec-6411-4eae-9a34-94ede034cf88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f916d574-hdv2g\" (UID: \"c082cbec-6411-4eae-9a34-94ede034cf88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"
Sep 16 12:19:37.613363 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.613342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c082cbec-6411-4eae-9a34-94ede034cf88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f916d574-hdv2g\" (UID: \"c082cbec-6411-4eae-9a34-94ede034cf88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"
Sep 16 12:19:37.660758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.660723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"
Sep 16 12:19:37.914620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.914587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:37.928251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:37.928174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"]
Sep 16 12:19:37.930624 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:19:37.930592    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc082cbec_6411_4eae_9a34_94ede034cf88.slice/crio-9822bed71c7ed38e595d0660d470e022a936682cf429487a940915375611aa76 WatchSource:0}: Error finding container 9822bed71c7ed38e595d0660d470e022a936682cf429487a940915375611aa76: Status 404 returned error can't find the container with id 9822bed71c7ed38e595d0660d470e022a936682cf429487a940915375611aa76
Sep 16 12:19:38.161257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.161229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g" event={"ID":"c082cbec-6411-4eae-9a34-94ede034cf88","Type":"ContainerStarted","Data":"9822bed71c7ed38e595d0660d470e022a936682cf429487a940915375611aa76"}
Sep 16 12:19:38.176882 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.176784    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g" podStartSLOduration=1.176769109 podStartE2EDuration="1.176769109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:38.174784159 +0000 UTC m=+1065.586801054" watchObservedRunningTime="2026-09-16 12:19:38.176769109 +0000 UTC m=+1065.588786004"
Sep 16 12:19:38.677555 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.677520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:38.843620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.843581    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"]
Sep 16 12:19:38.845907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.845887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:38.848521 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.848499    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"kube-root-ca.crt\""
Sep 16 12:19:38.849744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.849720    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"openshift-service-ca.crt\""
Sep 16 12:19:38.849885 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.849866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"default-dockercfg-46bj7\""
Sep 16 12:19:38.849942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.849872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-556077\"/\"e2e-conflict-b-067c87d4-kserve-self-signed-certs\""
Sep 16 12:19:38.855869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.855847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"]
Sep 16 12:19:38.925847 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.925808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-home\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:38.926052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.925862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-model-cache\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:38.926052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.925949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-tmp-dir\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:38.926052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.926024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdhc\" (UniqueName: \"kubernetes.io/projected/e54a176d-1a98-49b3-99ff-337f28adc0c9-kube-api-access-ckdhc\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:38.926242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.926077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-dshm\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:38.926242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.926131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a176d-1a98-49b3-99ff-337f28adc0c9-tls-certs\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:38.953746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:38.953696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:39.027159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-dshm\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a176d-1a98-49b3-99ff-337f28adc0c9-tls-certs\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-home\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-model-cache\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-tmp-dir\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdhc\" (UniqueName: \"kubernetes.io/projected/e54a176d-1a98-49b3-99ff-337f28adc0c9-kube-api-access-ckdhc\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027714 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-model-cache\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027798 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-home\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.027850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.027808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-tmp-dir\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.029589 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.029554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-dshm\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.029768 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.029751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a176d-1a98-49b3-99ff-337f28adc0c9-tls-certs\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.034749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.034723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdhc\" (UniqueName: \"kubernetes.io/projected/e54a176d-1a98-49b3-99ff-337f28adc0c9-kube-api-access-ckdhc\") pod \"e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") " pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.155615 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.155558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:39.169385 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.169349    2577 generic.go:356] "Generic (PLEG): container finished" podID="c082cbec-6411-4eae-9a34-94ede034cf88" containerID="370c465f1fed346f25c3bc3b1ff02af8bd0dba330e14a84ea7cf4cfb2f8fe196" exitCode=0
Sep 16 12:19:39.169529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.169457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g" event={"ID":"c082cbec-6411-4eae-9a34-94ede034cf88","Type":"ContainerDied","Data":"370c465f1fed346f25c3bc3b1ff02af8bd0dba330e14a84ea7cf4cfb2f8fe196"}
Sep 16 12:19:39.310233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.310198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:39.427079 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.427046    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"]
Sep 16 12:19:39.428485 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:19:39.428457    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode54a176d_1a98_49b3_99ff_337f28adc0c9.slice/crio-92f6a9309d25aafeae0b6493b35bb5e5eb6bb6039444a82c4253697aa44509dc WatchSource:0}: Error finding container 92f6a9309d25aafeae0b6493b35bb5e5eb6bb6039444a82c4253697aa44509dc: Status 404 returned error can't find the container with id 92f6a9309d25aafeae0b6493b35bb5e5eb6bb6039444a82c4253697aa44509dc
Sep 16 12:19:39.970708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:39.970674    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:40.176105 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.176063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584" event={"ID":"e54a176d-1a98-49b3-99ff-337f28adc0c9","Type":"ContainerStarted","Data":"6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc"}
Sep 16 12:19:40.176105 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.176107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584" event={"ID":"e54a176d-1a98-49b3-99ff-337f28adc0c9","Type":"ContainerStarted","Data":"92f6a9309d25aafeae0b6493b35bb5e5eb6bb6039444a82c4253697aa44509dc"}
Sep 16 12:19:40.192371 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.192301    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584" podStartSLOduration=2.19228165 podStartE2EDuration="2.19228165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:40.191515011 +0000 UTC m=+1067.603531907" watchObservedRunningTime="2026-09-16 12:19:40.19228165 +0000 UTC m=+1067.604298548"
Sep 16 12:19:40.309250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.309222    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"
Sep 16 12:19:40.439464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.439434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c082cbec-6411-4eae-9a34-94ede034cf88-maas-api-service-ca\") pod \"c082cbec-6411-4eae-9a34-94ede034cf88\" (UID: \"c082cbec-6411-4eae-9a34-94ede034cf88\") "
Sep 16 12:19:40.439817 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.439793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c082cbec-6411-4eae-9a34-94ede034cf88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c082cbec-6411-4eae-9a34-94ede034cf88" (UID: "c082cbec-6411-4eae-9a34-94ede034cf88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:19:40.540322 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.540234    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c082cbec-6411-4eae-9a34-94ede034cf88-maas-api-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:40.585667 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:40.585633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:41.135439 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.135404    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"]
Sep 16 12:19:41.135946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.135929    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c082cbec-6411-4eae-9a34-94ede034cf88" containerName="revoke-keys"
Sep 16 12:19:41.135946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.135947    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c082cbec-6411-4eae-9a34-94ede034cf88" containerName="revoke-keys"
Sep 16 12:19:41.136037 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.136028    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c082cbec-6411-4eae-9a34-94ede034cf88" containerName="revoke-keys"
Sep 16 12:19:41.138431 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.138405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.141196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.141171    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c26b8c-openshift-default-dockercfg-8s4cb\""
Sep 16 12:19:41.149408 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.149325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"]
Sep 16 12:19:41.160453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.160429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:41.183528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.183503    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"
Sep 16 12:19:41.183666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.183553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g" event={"ID":"c082cbec-6411-4eae-9a34-94ede034cf88","Type":"ContainerDied","Data":"9822bed71c7ed38e595d0660d470e022a936682cf429487a940915375611aa76"}
Sep 16 12:19:41.183666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.183610    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9822bed71c7ed38e595d0660d470e022a936682cf429487a940915375611aa76"
Sep 16 12:19:41.246577 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.246534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-data\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.246762 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.246725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-istio-token\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.246830 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.246765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-credential-socket\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.246830 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.246803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-socket\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.246944 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.246839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-certs\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.246998 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.246944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-envoy\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.246998 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.246981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319f309b-440b-4ab9-89ed-16e5de33736a-istiod-ca-cert\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.247106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.247025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319f309b-440b-4ab9-89ed-16e5de33736a-istio-podinfo\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.247459 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.247429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxl6\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-kube-api-access-fnxl6\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.278775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.278745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:41.348426 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-data\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-istio-token\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-credential-socket\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-socket\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-certs\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-envoy\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319f309b-440b-4ab9-89ed-16e5de33736a-istiod-ca-cert\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348977 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319f309b-440b-4ab9-89ed-16e5de33736a-istio-podinfo\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348977 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxl6\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-kube-api-access-fnxl6\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348977 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-data\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348977 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-certs\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.348977 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.348958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-credential-socket\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.349471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.349431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-socket\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.350046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.350001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319f309b-440b-4ab9-89ed-16e5de33736a-istiod-ca-cert\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.351883 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.351858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-envoy\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.352071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.352054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319f309b-440b-4ab9-89ed-16e5de33736a-istio-podinfo\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.357171 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.357147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-istio-token\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.357278 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.357147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxl6\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-kube-api-access-fnxl6\") pod \"e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") " pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.375634 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.375552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:41.450187 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.450157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:41.721934 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.721881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"]
Sep 16 12:19:41.724082 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:19:41.724046    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319f309b_440b_4ab9_89ed_16e5de33736a.slice/crio-f5e0bc660b87569d903af32f0768ed974a65c40e4cb3acfb074547cf550b2798 WatchSource:0}: Error finding container f5e0bc660b87569d903af32f0768ed974a65c40e4cb3acfb074547cf550b2798: Status 404 returned error can't find the container with id f5e0bc660b87569d903af32f0768ed974a65c40e4cb3acfb074547cf550b2798
Sep 16 12:19:41.726560 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.726521    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:19:41.726690 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.726622    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:19:41.726690 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.726655    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:19:41.974013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:41.973926    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:42.159422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.159388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:42.191674 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.191643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm" event={"ID":"319f309b-440b-4ab9-89ed-16e5de33736a","Type":"ContainerStarted","Data":"5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163"}
Sep 16 12:19:42.191847 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.191697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm" event={"ID":"319f309b-440b-4ab9-89ed-16e5de33736a","Type":"ContainerStarted","Data":"f5e0bc660b87569d903af32f0768ed974a65c40e4cb3acfb074547cf550b2798"}
Sep 16 12:19:42.211868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.211814    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm" podStartSLOduration=1.211797643 podStartE2EDuration="1.211797643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:19:42.209739846 +0000 UTC m=+1069.621756742" watchObservedRunningTime="2026-09-16 12:19:42.211797643 +0000 UTC m=+1069.623814539"
Sep 16 12:19:42.450838 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.450805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:42.455762 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.455736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:42.526880 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.526836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:42.620512 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.620475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:42.807872 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:42.807795    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:43.196787 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:43.196756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:43.197755 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:43.197732    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:19:44.999671 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:44.999636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:19:47.477038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:47.477006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:48.010952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.010923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:48.034731 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.034696    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"]
Sep 16 12:19:48.034996 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.034944    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx" podUID="7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" containerName="payload-processing" containerID="cri-o://b8196aaa896b2c8f66589a36d9873e3e7e978a2f1992d5591a4991d735246bce" gracePeriod=30
Sep 16 12:19:48.041005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.040976    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"]
Sep 16 12:19:48.041214 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.041192    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5" podUID="e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" containerName="payload-pre-processing" containerID="cri-o://35b198a65fe6714f5fcba3e02008a0231fd33d7390657ed5031b534ef097badb" gracePeriod=30
Sep 16 12:19:48.078270 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.078240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:48.144209 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.144175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:48.195854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.195814    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"]
Sep 16 12:19:48.201756 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.201723    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f916d574-hdv2g"]
Sep 16 12:19:48.228888 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.228861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:48.229604 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.229560    2577 generic.go:356] "Generic (PLEG): container finished" podID="7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" containerID="b8196aaa896b2c8f66589a36d9873e3e7e978a2f1992d5591a4991d735246bce" exitCode=0
Sep 16 12:19:48.229715 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.229603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx" event={"ID":"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5","Type":"ContainerDied","Data":"b8196aaa896b2c8f66589a36d9873e3e7e978a2f1992d5591a4991d735246bce"}
Sep 16 12:19:48.231411 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.231391    2577 generic.go:356] "Generic (PLEG): container finished" podID="e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" containerID="35b198a65fe6714f5fcba3e02008a0231fd33d7390657ed5031b534ef097badb" exitCode=0
Sep 16 12:19:48.231507 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.231432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5" event={"ID":"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9","Type":"ContainerDied","Data":"35b198a65fe6714f5fcba3e02008a0231fd33d7390657ed5031b534ef097badb"}
Sep 16 12:19:48.343291 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.343261    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:19:48.346157 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.346133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:19:48.514994 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.514959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9p4r\" (UniqueName: \"kubernetes.io/projected/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-kube-api-access-q9p4r\") pod \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") "
Sep 16 12:19:48.515457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.515008    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjn9h\" (UniqueName: \"kubernetes.io/projected/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-kube-api-access-kjn9h\") pod \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") "
Sep 16 12:19:48.515457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.515067    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-plugins-config-volume\") pod \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\" (UID: \"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9\") "
Sep 16 12:19:48.515457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.515141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-plugins-config-volume\") pod \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\" (UID: \"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5\") "
Sep 16 12:19:48.515606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.515546    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" (UID: "7cbbef58-2f00-4c9a-a24e-fd6b547a97c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:19:48.515606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.515551    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" (UID: "e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:19:48.517110 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.517085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-kube-api-access-q9p4r" (OuterVolumeSpecName: "kube-api-access-q9p4r") pod "7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" (UID: "7cbbef58-2f00-4c9a-a24e-fd6b547a97c5"). InnerVolumeSpecName "kube-api-access-q9p4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:48.517183 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.517106    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-kube-api-access-kjn9h" (OuterVolumeSpecName: "kube-api-access-kjn9h") pod "e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" (UID: "e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9"). InnerVolumeSpecName "kube-api-access-kjn9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:19:48.580868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.580771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:48.616730 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.616696    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.616730 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.616727    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9p4r\" (UniqueName: \"kubernetes.io/projected/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5-kube-api-access-q9p4r\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.616730 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.616736    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjn9h\" (UniqueName: \"kubernetes.io/projected/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-kube-api-access-kjn9h\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.616967 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.616745    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:19:48.706196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:48.706165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:49.112934 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.112891    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c082cbec-6411-4eae-9a34-94ede034cf88" path="/var/lib/kubelet/pods/c082cbec-6411-4eae-9a34-94ede034cf88/volumes"
Sep 16 12:19:49.155873 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.155832    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:49.155873 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.155877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:49.168514 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.168485    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:49.237775 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.237745    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"
Sep 16 12:19:49.237957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.237782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx" event={"ID":"7cbbef58-2f00-4c9a-a24e-fd6b547a97c5","Type":"ContainerDied","Data":"465ab856b5d6a404886406f058f14f8e2d96068a08306fca8095cd765bdc1fe0"}
Sep 16 12:19:49.237957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.237853    2577 scope.go:117] "RemoveContainer" containerID="b8196aaa896b2c8f66589a36d9873e3e7e978a2f1992d5591a4991d735246bce"
Sep 16 12:19:49.239323 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.239176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5" event={"ID":"e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9","Type":"ContainerDied","Data":"42ab2c3422b6aad3fee04f40f1c805d699d66b0009f19674dfc5359b7960c506"}
Sep 16 12:19:49.239323 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.239206    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"
Sep 16 12:19:49.247381 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.247355    2577 scope.go:117] "RemoveContainer" containerID="35b198a65fe6714f5fcba3e02008a0231fd33d7390657ed5031b534ef097badb"
Sep 16 12:19:49.253031 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.253012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:19:49.262359 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.262338    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"]
Sep 16 12:19:49.268173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.268152    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f916d574-6756588c4-7r9h5"]
Sep 16 12:19:49.282458 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.282434    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"]
Sep 16 12:19:49.289666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.289643    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f916d574-579c745c69-6nmbx"]
Sep 16 12:19:49.292102 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.292083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:49.976243 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:49.976215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:50.643486 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:50.643452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:51.111997 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:51.111922    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" path="/var/lib/kubelet/pods/7cbbef58-2f00-4c9a-a24e-fd6b547a97c5/volumes"
Sep 16 12:19:51.112327 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:51.112271    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" path="/var/lib/kubelet/pods/e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9/volumes"
Sep 16 12:19:51.384843 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:51.384813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:53.084974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:53.084936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:53.185882 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:53.185859    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:54.441537 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:54.441497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:54.541279 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:54.541238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:54.759358 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:54.759269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:55.296937 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:55.296903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:55.392666 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:55.392624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:19:55.535957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:19:55.535925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:01.291897 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:01.291863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:01.393656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:01.393617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:01.493031 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:01.492999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:01.686354 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:01.686318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:07.224968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.224906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:07.371327 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.371287    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"]
Sep 16 12:20:07.371808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.371786    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" containerName="payload-pre-processing"
Sep 16 12:20:07.371808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.371807    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" containerName="payload-pre-processing"
Sep 16 12:20:07.371992 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.371826    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" containerName="payload-processing"
Sep 16 12:20:07.371992 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.371833    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" containerName="payload-processing"
Sep 16 12:20:07.371992 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.371937    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cbbef58-2f00-4c9a-a24e-fd6b547a97c5" containerName="payload-processing"
Sep 16 12:20:07.371992 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.371957    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a5e9de-fe6e-4b7e-896e-26ec9a9a64e9" containerName="payload-pre-processing"
Sep 16 12:20:07.377260 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.377234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"
Sep 16 12:20:07.380857 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.380829    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9lc86\""
Sep 16 12:20:07.381104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.380847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:20:07.382591 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.382540    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"]
Sep 16 12:20:07.465158 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.465117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d693db7-ec29-496c-8467-a9d6212f1f72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-641983-gk69k\" (UID: \"4d693db7-ec29-496c-8467-a9d6212f1f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"
Sep 16 12:20:07.566172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.566085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d693db7-ec29-496c-8467-a9d6212f1f72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-641983-gk69k\" (UID: \"4d693db7-ec29-496c-8467-a9d6212f1f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"
Sep 16 12:20:07.566758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.566737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d693db7-ec29-496c-8467-a9d6212f1f72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-641983-gk69k\" (UID: \"4d693db7-ec29-496c-8467-a9d6212f1f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"
Sep 16 12:20:07.690662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.690624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"
Sep 16 12:20:07.775335 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.775276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:07.961464 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:07.961436    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"]
Sep 16 12:20:07.963156 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:20:07.963119    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d693db7_ec29_496c_8467_a9d6212f1f72.slice/crio-96f6826312f41a620f1af9eb5837d7101121d0185adc47df92bfba34898f35f2 WatchSource:0}: Error finding container 96f6826312f41a620f1af9eb5837d7101121d0185adc47df92bfba34898f35f2: Status 404 returned error can't find the container with id 96f6826312f41a620f1af9eb5837d7101121d0185adc47df92bfba34898f35f2
Sep 16 12:20:08.342010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:08.341976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:08.348285 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:08.348253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k" event={"ID":"4d693db7-ec29-496c-8467-a9d6212f1f72","Type":"ContainerStarted","Data":"f8bb0691d43e07e60dc02f7a82b02a1a612faf4d91f60869c62593a5a8bc2ff3"}
Sep 16 12:20:08.348494 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:08.348315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k" event={"ID":"4d693db7-ec29-496c-8467-a9d6212f1f72","Type":"ContainerStarted","Data":"96f6826312f41a620f1af9eb5837d7101121d0185adc47df92bfba34898f35f2"}
Sep 16 12:20:08.365359 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:08.365290    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k" podStartSLOduration=1.365268506 podStartE2EDuration="1.365268506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:08.364467115 +0000 UTC m=+1095.776484012" watchObservedRunningTime="2026-09-16 12:20:08.365268506 +0000 UTC m=+1095.777285404"
Sep 16 12:20:08.443991 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:08.443822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:08.807162 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:08.807127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.356231 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.356191    2577 generic.go:356] "Generic (PLEG): container finished" podID="4d693db7-ec29-496c-8467-a9d6212f1f72" containerID="f8bb0691d43e07e60dc02f7a82b02a1a612faf4d91f60869c62593a5a8bc2ff3" exitCode=0
Sep 16 12:20:09.356825 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.356380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k" event={"ID":"4d693db7-ec29-496c-8467-a9d6212f1f72","Type":"ContainerDied","Data":"f8bb0691d43e07e60dc02f7a82b02a1a612faf4d91f60869c62593a5a8bc2ff3"}
Sep 16 12:20:09.514102 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.514059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.688887 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.688847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.940088 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.939999    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"]
Sep 16 12:20:09.944126 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.944100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.947159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.947131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-867202-openshift-default-dockercfg-kbhjs\""
Sep 16 12:20:09.956622 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.956597    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"]
Sep 16 12:20:09.967129 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.967096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:09.981844 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.981804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-socket\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.981860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istiod-ca-cert\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.981929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-certs\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.981965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-podinfo\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.982024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-data\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.982057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-envoy\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982343 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.982090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-token\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982343 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.982120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxp5\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-kube-api-access-qlxp5\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:09.982343 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:09.982157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-credential-socket\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.082605 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.082550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-credential-socket\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.082812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.082740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-socket\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.082812 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.082794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istiod-ca-cert\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.082937 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.082846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-certs\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.082937 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.082883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-podinfo\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.082965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-data\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-envoy\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-socket\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-token\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxp5\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-kube-api-access-qlxp5\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083295 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-certs\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083468 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-credential-socket\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.083802 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istiod-ca-cert\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.084015 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.083843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-data\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.085952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.085923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-envoy\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.086383 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.086363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-podinfo\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.091173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.091149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-token\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.091523 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.091499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxp5\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-kube-api-access-qlxp5\") pod \"e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") " pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.255173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.255073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:10.434441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.434405    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:10.506878 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.506814    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"
Sep 16 12:20:10.544076 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.544025    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"]
Sep 16 12:20:10.544348 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:20:10.544313    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dd50c1_e7b9_44be_ab2f_2ad8d8588327.slice/crio-89cb2adf366f3dd0a6e1e8f52d00c743b89cb9336bf0c1f368badd8901a29349 WatchSource:0}: Error finding container 89cb2adf366f3dd0a6e1e8f52d00c743b89cb9336bf0c1f368badd8901a29349: Status 404 returned error can't find the container with id 89cb2adf366f3dd0a6e1e8f52d00c743b89cb9336bf0c1f368badd8901a29349
Sep 16 12:20:10.544348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.544317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:10.547035 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.547003    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:20:10.547121 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.547065    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:20:10.547121 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.547094    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:20:10.588223 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.588190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d693db7-ec29-496c-8467-a9d6212f1f72-maas-api-service-ca\") pod \"4d693db7-ec29-496c-8467-a9d6212f1f72\" (UID: \"4d693db7-ec29-496c-8467-a9d6212f1f72\") "
Sep 16 12:20:10.588696 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.588659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d693db7-ec29-496c-8467-a9d6212f1f72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d693db7-ec29-496c-8467-a9d6212f1f72" (UID: "4d693db7-ec29-496c-8467-a9d6212f1f72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:10.588866 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:10.588832    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d693db7-ec29-496c-8467-a9d6212f1f72-maas-api-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:20:11.370235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:11.370141    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"
Sep 16 12:20:11.370235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:11.370172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k" event={"ID":"4d693db7-ec29-496c-8467-a9d6212f1f72","Type":"ContainerDied","Data":"96f6826312f41a620f1af9eb5837d7101121d0185adc47df92bfba34898f35f2"}
Sep 16 12:20:11.370235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:11.370222    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f6826312f41a620f1af9eb5837d7101121d0185adc47df92bfba34898f35f2"
Sep 16 12:20:11.372256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:11.372163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" event={"ID":"48dd50c1-e7b9-44be-ab2f-2ad8d8588327","Type":"ContainerStarted","Data":"0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84"}
Sep 16 12:20:11.372256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:11.372218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" event={"ID":"48dd50c1-e7b9-44be-ab2f-2ad8d8588327","Type":"ContainerStarted","Data":"89cb2adf366f3dd0a6e1e8f52d00c743b89cb9336bf0c1f368badd8901a29349"}
Sep 16 12:20:11.399514 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:11.399484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:11.400230 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:11.400188    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podStartSLOduration=2.400173009 podStartE2EDuration="2.400173009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:11.398285542 +0000 UTC m=+1098.810302441" watchObservedRunningTime="2026-09-16 12:20:11.400173009 +0000 UTC m=+1098.812189904"
Sep 16 12:20:12.107833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:12.107801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:12.256167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:12.256120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:12.261619 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:12.261593    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:12.383461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:12.383429    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:12.384407 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:12.384385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:12.977163 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:12.977117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:13.513143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:13.513108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:13.598782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:13.598743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:13.965058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:13.965024    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:14.279408 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:14.279315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:14.753700 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:14.753666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:15.382793 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:15.382759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.175014 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.174969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.901064 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.901034    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"]
Sep 16 12:20:16.901431 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.901413    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d693db7-ec29-496c-8467-a9d6212f1f72" containerName="revoke-keys"
Sep 16 12:20:16.901431 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.901429    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d693db7-ec29-496c-8467-a9d6212f1f72" containerName="revoke-keys"
Sep 16 12:20:16.901636 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.901489    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d693db7-ec29-496c-8467-a9d6212f1f72" containerName="revoke-keys"
Sep 16 12:20:16.905732 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.905706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:16.909610 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.909588    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:20:16.910656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.910636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:20:16.910849 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.910833    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-867202-serving-cert\""
Sep 16 12:20:16.910932 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.910848    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7fbn\""
Sep 16 12:20:16.910992 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.910936    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:20:16.917694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.917663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"]
Sep 16 12:20:16.940929 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.940894    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:16.944131 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.944102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:16.944208 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.944149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:16.944208 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.944196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:16.944311 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:16.944294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgds\" (UniqueName: \"kubernetes.io/projected/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-kube-api-access-rhgds\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.041175 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.041141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:17.045440 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.045414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.045558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.045458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.045558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.045478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.045558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.045532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgds\" (UniqueName: \"kubernetes.io/projected/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-kube-api-access-rhgds\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.045741 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:20:17.045593    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-serving-cert: secret "maas-api-e2e-shared-a-w3-867202-serving-cert" not found
Sep 16 12:20:17.045741 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:20:17.045654    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls podName:fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05 nodeName:}" failed. No retries permitted until 2026-09-16 12:20:17.545638418 +0000 UTC m=+1104.957655306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls") pod "maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh" (UID: "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05") : secret "maas-api-e2e-shared-a-w3-867202-serving-cert" not found
Sep 16 12:20:17.046152 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.046129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.047930 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.047910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.062459 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.062419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgds\" (UniqueName: \"kubernetes.io/projected/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-kube-api-access-rhgds\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.141944 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.141905    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:17.409206 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.409168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:17.507379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.507344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:17.550482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.550438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.552940 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.552912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:17.785477 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.785380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:17.817421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:17.817384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:18.089321 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:18.089232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:20:18.098553 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:18.098518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"]
Sep 16 12:20:18.100885 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:20:18.100856    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6e5e5c_bcd7_4b2c_a5ac_8ff60a480c05.slice/crio-90f539646f70991a373bcd086a33bba6551ea811aac978751bfe6eafb22fda25 WatchSource:0}: Error finding container 90f539646f70991a373bcd086a33bba6551ea811aac978751bfe6eafb22fda25: Status 404 returned error can't find the container with id 90f539646f70991a373bcd086a33bba6551ea811aac978751bfe6eafb22fda25
Sep 16 12:20:18.102648 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:18.102628    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:20:18.417776 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:18.417730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh" event={"ID":"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05","Type":"ContainerStarted","Data":"90f539646f70991a373bcd086a33bba6551ea811aac978751bfe6eafb22fda25"}
Sep 16 12:20:18.478957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:18.478902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:19.023158 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.023110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:19.045455 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.045422    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"]
Sep 16 12:20:19.045661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.045641    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw" podUID="82c855a2-9e46-448a-b01f-1cc4205de1c4" containerName="payload-processing" containerID="cri-o://ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5" gracePeriod=30
Sep 16 12:20:19.122993 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.122954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:19.224453 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.224422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:19.225522 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.225496    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"]
Sep 16 12:20:19.229296 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.229274    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-641983-gk69k"]
Sep 16 12:20:19.300139 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.300109    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:20:19.321678 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.321642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:19.370994 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.370949    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbsxv\" (UniqueName: \"kubernetes.io/projected/82c855a2-9e46-448a-b01f-1cc4205de1c4-kube-api-access-rbsxv\") pod \"82c855a2-9e46-448a-b01f-1cc4205de1c4\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") "
Sep 16 12:20:19.371242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.371056    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c855a2-9e46-448a-b01f-1cc4205de1c4-plugins-config-volume\") pod \"82c855a2-9e46-448a-b01f-1cc4205de1c4\" (UID: \"82c855a2-9e46-448a-b01f-1cc4205de1c4\") "
Sep 16 12:20:19.371483 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.371450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c855a2-9e46-448a-b01f-1cc4205de1c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82c855a2-9e46-448a-b01f-1cc4205de1c4" (UID: "82c855a2-9e46-448a-b01f-1cc4205de1c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:19.374201 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.373725    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c855a2-9e46-448a-b01f-1cc4205de1c4-kube-api-access-rbsxv" (OuterVolumeSpecName: "kube-api-access-rbsxv") pod "82c855a2-9e46-448a-b01f-1cc4205de1c4" (UID: "82c855a2-9e46-448a-b01f-1cc4205de1c4"). InnerVolumeSpecName "kube-api-access-rbsxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:19.426790 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.426758    2577 generic.go:356] "Generic (PLEG): container finished" podID="82c855a2-9e46-448a-b01f-1cc4205de1c4" containerID="ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5" exitCode=0
Sep 16 12:20:19.427257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.426816    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"
Sep 16 12:20:19.427257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.426843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw" event={"ID":"82c855a2-9e46-448a-b01f-1cc4205de1c4","Type":"ContainerDied","Data":"ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5"}
Sep 16 12:20:19.427257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.426899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw" event={"ID":"82c855a2-9e46-448a-b01f-1cc4205de1c4","Type":"ContainerDied","Data":"f53c540425c54d35be7027cb5c63ed6c66b19ce746e3c5d2f5c0c004333813b2"}
Sep 16 12:20:19.427257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.426927    2577 scope.go:117] "RemoveContainer" containerID="ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5"
Sep 16 12:20:19.428991 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.428961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh" event={"ID":"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05","Type":"ContainerStarted","Data":"6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed"}
Sep 16 12:20:19.429210 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.429137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:19.438978 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.438957    2577 scope.go:117] "RemoveContainer" containerID="ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5"
Sep 16 12:20:19.439338 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:20:19.439315    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5\": container with ID starting with ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5 not found: ID does not exist" containerID="ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5"
Sep 16 12:20:19.439407 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.439353    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5"} err="failed to get container status \"ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5\": rpc error: code = NotFound desc = could not find container \"ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5\": container with ID starting with ac70897ec41d839fd3fbbdb3bd061a2ae4144fbccb5db97ba86657897da1a5f5 not found: ID does not exist"
Sep 16 12:20:19.453637 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.453516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh" podStartSLOduration=3.068210846 podStartE2EDuration="3.453499862s" totalImagesPullingTime="385.289016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:19.452466516 +0000 UTC m=+1106.864483411" watchObservedRunningTime="2026-09-16 12:20:19.453499862 +0000 UTC m=+1106.865516757"
Sep 16 12:20:19.469314 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.469280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"]
Sep 16 12:20:19.472454 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.472421    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82c855a2-9e46-448a-b01f-1cc4205de1c4-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:20:19.472454 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.472453    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbsxv\" (UniqueName: \"kubernetes.io/projected/82c855a2-9e46-448a-b01f-1cc4205de1c4-kube-api-access-rbsxv\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:20:19.473189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.473161    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-641983-5b8954568c-qq5bw"]
Sep 16 12:20:19.477170 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.477148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:19.672379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:19.672342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:20.248148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:20.248105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.089288 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:21.089253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:21.111446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:21.111411    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d693db7-ec29-496c-8467-a9d6212f1f72" path="/var/lib/kubelet/pods/4d693db7-ec29-496c-8467-a9d6212f1f72/volumes"
Sep 16 12:20:21.111868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:21.111851    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c855a2-9e46-448a-b01f-1cc4205de1c4" path="/var/lib/kubelet/pods/82c855a2-9e46-448a-b01f-1cc4205de1c4/volumes"
Sep 16 12:20:21.775728 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:21.775689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:22.444099 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:22.444069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:23.157800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:23.157764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:25.441710 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:25.441677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:20:25.501227 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:25.501189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:25.572588 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:25.572544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:25.811777 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:25.811684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:26.363803 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:26.363774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:27.218052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:27.218020    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:27.877552 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:27.877517    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:28.583376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:28.583339    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:29.221211 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:29.221175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:30.075179 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:30.075146    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:30.793809 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:30.793775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:31.499221 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:31.499189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:32.281212 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:32.281180    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:32.985435 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:32.985401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:33.374403 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:33.374318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:33.711964 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:33.711933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:33.812694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:33.812665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:34.061342 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:34.061251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:34.347161 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:34.347058    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:36.083364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:36.083329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:36.769305 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:36.769223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:36.897175 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:36.897135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:37.070367 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:37.070275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:37.765539 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:37.765499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:38.571547 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:38.571502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.128370 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.128308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.150978 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.150944    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"]
Sep 16 12:20:39.151420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.151383    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d" podUID="8bff3188-5f14-4c25-8534-1ae8b4c48b59" containerName="payload-processing" containerID="cri-o://acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4" gracePeriod=30
Sep 16 12:20:39.160605 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.160551    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"]
Sep 16 12:20:39.161334 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.161290    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs" podUID="2268e65e-0da8-4714-af81-385c06123bb1" containerName="payload-pre-processing" containerID="cri-o://348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7" gracePeriod=30
Sep 16 12:20:39.195868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.195833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.252929 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.252897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.451092 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.450543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.483813 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.483537    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:20:39.491361 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.491204    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:20:39.524657 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.524389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.548773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.548073    2577 generic.go:356] "Generic (PLEG): container finished" podID="2268e65e-0da8-4714-af81-385c06123bb1" containerID="348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7" exitCode=0
Sep 16 12:20:39.548773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.548165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs" event={"ID":"2268e65e-0da8-4714-af81-385c06123bb1","Type":"ContainerDied","Data":"348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7"}
Sep 16 12:20:39.548773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.548246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs" event={"ID":"2268e65e-0da8-4714-af81-385c06123bb1","Type":"ContainerDied","Data":"648be7c830b0e6478500d05a2348a8c2aaf7884ee75395319f408e2da8566d57"}
Sep 16 12:20:39.548773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.548284    2577 scope.go:117] "RemoveContainer" containerID="348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7"
Sep 16 12:20:39.548773 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.548429    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"
Sep 16 12:20:39.551042 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.551014    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bff3188-5f14-4c25-8534-1ae8b4c48b59" containerID="acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4" exitCode=0
Sep 16 12:20:39.551181 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.551118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d" event={"ID":"8bff3188-5f14-4c25-8534-1ae8b4c48b59","Type":"ContainerDied","Data":"acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4"}
Sep 16 12:20:39.551181 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.551156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d" event={"ID":"8bff3188-5f14-4c25-8534-1ae8b4c48b59","Type":"ContainerDied","Data":"b8160be48ce0bf79521cc2515df062fb95ff18d80b61cdcc7f3e5b120424d382"}
Sep 16 12:20:39.551305 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.551231    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"
Sep 16 12:20:39.561375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.561343    2577 scope.go:117] "RemoveContainer" containerID="348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7"
Sep 16 12:20:39.561675 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:20:39.561648    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7\": container with ID starting with 348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7 not found: ID does not exist" containerID="348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7"
Sep 16 12:20:39.561752 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.561686    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7"} err="failed to get container status \"348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7\": rpc error: code = NotFound desc = could not find container \"348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7\": container with ID starting with 348d3ff8b962cc2674dc602376c44d8cd5ff9a756b74ca7d327473ca6371ccc7 not found: ID does not exist"
Sep 16 12:20:39.561752 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.561710    2577 scope.go:117] "RemoveContainer" containerID="acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4"
Sep 16 12:20:39.568797 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.568770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2268e65e-0da8-4714-af81-385c06123bb1-plugins-config-volume\") pod \"2268e65e-0da8-4714-af81-385c06123bb1\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") "
Sep 16 12:20:39.568912 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.568842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bff3188-5f14-4c25-8534-1ae8b4c48b59-plugins-config-volume\") pod \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") "
Sep 16 12:20:39.568988 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.568916    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fnwt\" (UniqueName: \"kubernetes.io/projected/2268e65e-0da8-4714-af81-385c06123bb1-kube-api-access-8fnwt\") pod \"2268e65e-0da8-4714-af81-385c06123bb1\" (UID: \"2268e65e-0da8-4714-af81-385c06123bb1\") "
Sep 16 12:20:39.568988 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.568973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kj2b\" (UniqueName: \"kubernetes.io/projected/8bff3188-5f14-4c25-8534-1ae8b4c48b59-kube-api-access-2kj2b\") pod \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\" (UID: \"8bff3188-5f14-4c25-8534-1ae8b4c48b59\") "
Sep 16 12:20:39.569480 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.569451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bff3188-5f14-4c25-8534-1ae8b4c48b59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bff3188-5f14-4c25-8534-1ae8b4c48b59" (UID: "8bff3188-5f14-4c25-8534-1ae8b4c48b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:39.569657 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.569581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2268e65e-0da8-4714-af81-385c06123bb1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2268e65e-0da8-4714-af81-385c06123bb1" (UID: "2268e65e-0da8-4714-af81-385c06123bb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:39.571391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.571360    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bff3188-5f14-4c25-8534-1ae8b4c48b59-kube-api-access-2kj2b" (OuterVolumeSpecName: "kube-api-access-2kj2b") pod "8bff3188-5f14-4c25-8534-1ae8b4c48b59" (UID: "8bff3188-5f14-4c25-8534-1ae8b4c48b59"). InnerVolumeSpecName "kube-api-access-2kj2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:39.571905 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.571882    2577 scope.go:117] "RemoveContainer" containerID="acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4"
Sep 16 12:20:39.572026 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.572002    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2268e65e-0da8-4714-af81-385c06123bb1-kube-api-access-8fnwt" (OuterVolumeSpecName: "kube-api-access-8fnwt") pod "2268e65e-0da8-4714-af81-385c06123bb1" (UID: "2268e65e-0da8-4714-af81-385c06123bb1"). InnerVolumeSpecName "kube-api-access-8fnwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:39.572298 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:20:39.572274    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4\": container with ID starting with acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4 not found: ID does not exist" containerID="acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4"
Sep 16 12:20:39.572349 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.572311    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4"} err="failed to get container status \"acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4\": rpc error: code = NotFound desc = could not find container \"acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4\": container with ID starting with acbb7d94b6aaa655bf3a35a638580f25dddf2ae66a188e3ce5c39b7de581bff4 not found: ID does not exist"
Sep 16 12:20:39.669946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.669859    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fnwt\" (UniqueName: \"kubernetes.io/projected/2268e65e-0da8-4714-af81-385c06123bb1-kube-api-access-8fnwt\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.669946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.669898    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kj2b\" (UniqueName: \"kubernetes.io/projected/8bff3188-5f14-4c25-8534-1ae8b4c48b59-kube-api-access-2kj2b\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.669946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.669914    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2268e65e-0da8-4714-af81-385c06123bb1-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.669946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.669927    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bff3188-5f14-4c25-8534-1ae8b4c48b59-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:20:39.707948 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.707914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.808308 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.808257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:39.874592 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.874533    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"]
Sep 16 12:20:39.876547 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.876521    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08170a-6b689bc8f8-6vqxs"]
Sep 16 12:20:39.889504 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.889477    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"]
Sep 16 12:20:39.894166 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.894139    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08170a-66447b7bb9-xkd7d"]
Sep 16 12:20:39.947267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:39.947184    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:40.036526 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.036482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:40.039027 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.038997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"]
Sep 16 12:20:40.039612 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039593    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2268e65e-0da8-4714-af81-385c06123bb1" containerName="payload-pre-processing"
Sep 16 12:20:40.039672 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039615    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2268e65e-0da8-4714-af81-385c06123bb1" containerName="payload-pre-processing"
Sep 16 12:20:40.039672 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039633    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bff3188-5f14-4c25-8534-1ae8b4c48b59" containerName="payload-processing"
Sep 16 12:20:40.039672 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039643    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bff3188-5f14-4c25-8534-1ae8b4c48b59" containerName="payload-processing"
Sep 16 12:20:40.039783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039689    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c855a2-9e46-448a-b01f-1cc4205de1c4" containerName="payload-processing"
Sep 16 12:20:40.039783 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039699    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c855a2-9e46-448a-b01f-1cc4205de1c4" containerName="payload-processing"
Sep 16 12:20:40.039848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039804    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2268e65e-0da8-4714-af81-385c06123bb1" containerName="payload-pre-processing"
Sep 16 12:20:40.039848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039818    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c855a2-9e46-448a-b01f-1cc4205de1c4" containerName="payload-processing"
Sep 16 12:20:40.039848 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.039834    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bff3188-5f14-4c25-8534-1ae8b4c48b59" containerName="payload-processing"
Sep 16 12:20:40.044808 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.044780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.047368 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.047344    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6f8610-dockercfg-xcj4h\""
Sep 16 12:20:40.047553 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.047533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6f8610\""
Sep 16 12:20:40.053001 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.052978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"]
Sep 16 12:20:40.083712 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.083676    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"]
Sep 16 12:20:40.087039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.087021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.098123 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.098095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"]
Sep 16 12:20:40.174477 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.174441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4p5x\" (UniqueName: \"kubernetes.io/projected/d18768f4-3c50-48c3-b532-0edf316c15d6-kube-api-access-r4p5x\") pod \"payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.174911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.174498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18768f4-3c50-48c3-b532-0edf316c15d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.174911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.174546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4x9\" (UniqueName: \"kubernetes.io/projected/a182a05f-ddfe-4248-a416-b84a058fe3a0-kube-api-access-zx4x9\") pod \"payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.174911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.174662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a182a05f-ddfe-4248-a416-b84a058fe3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.275969 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.275892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a182a05f-ddfe-4248-a416-b84a058fe3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.276101 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.276046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4p5x\" (UniqueName: \"kubernetes.io/projected/d18768f4-3c50-48c3-b532-0edf316c15d6-kube-api-access-r4p5x\") pod \"payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.276152 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.276122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18768f4-3c50-48c3-b532-0edf316c15d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.276203 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.276154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4x9\" (UniqueName: \"kubernetes.io/projected/a182a05f-ddfe-4248-a416-b84a058fe3a0-kube-api-access-zx4x9\") pod \"payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.276854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.276820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a182a05f-ddfe-4248-a416-b84a058fe3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.276985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.276885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18768f4-3c50-48c3-b532-0edf316c15d6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.284921 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.284845    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4p5x\" (UniqueName: \"kubernetes.io/projected/d18768f4-3c50-48c3-b532-0edf316c15d6-kube-api-access-r4p5x\") pod \"payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.285056 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.284906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4x9\" (UniqueName: \"kubernetes.io/projected/a182a05f-ddfe-4248-a416-b84a058fe3a0-kube-api-access-zx4x9\") pod \"payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.333974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.333932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:40.357409 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.357378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:40.399882 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.399839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:40.636679 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.636642    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"]
Sep 16 12:20:40.638276 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:20:40.638248    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda182a05f_ddfe_4248_a416_b84a058fe3a0.slice/crio-fc4e2f80cb02548ff6ec7bcf5fa76d81dc4b11b132d3cdc69db3d706d57a227f WatchSource:0}: Error finding container fc4e2f80cb02548ff6ec7bcf5fa76d81dc4b11b132d3cdc69db3d706d57a227f: Status 404 returned error can't find the container with id fc4e2f80cb02548ff6ec7bcf5fa76d81dc4b11b132d3cdc69db3d706d57a227f
Sep 16 12:20:40.691798 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:40.691761    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"]
Sep 16 12:20:40.692061 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:20:40.692037    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18768f4_3c50_48c3_b532_0edf316c15d6.slice/crio-17eb79033d36c2c6a1c5ad436a1eba22f628278b49d172b31d632840b3a2d41d WatchSource:0}: Error finding container 17eb79033d36c2c6a1c5ad436a1eba22f628278b49d172b31d632840b3a2d41d: Status 404 returned error can't find the container with id 17eb79033d36c2c6a1c5ad436a1eba22f628278b49d172b31d632840b3a2d41d
Sep 16 12:20:41.112726 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.112647    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2268e65e-0da8-4714-af81-385c06123bb1" path="/var/lib/kubelet/pods/2268e65e-0da8-4714-af81-385c06123bb1/volumes"
Sep 16 12:20:41.113026 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.113012    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bff3188-5f14-4c25-8534-1ae8b4c48b59" path="/var/lib/kubelet/pods/8bff3188-5f14-4c25-8534-1ae8b4c48b59/volumes"
Sep 16 12:20:41.201135 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.201104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:41.567643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.567607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc" event={"ID":"a182a05f-ddfe-4248-a416-b84a058fe3a0","Type":"ContainerStarted","Data":"380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d"}
Sep 16 12:20:41.569297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.567769    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:41.569297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.567800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc" event={"ID":"a182a05f-ddfe-4248-a416-b84a058fe3a0","Type":"ContainerStarted","Data":"fc4e2f80cb02548ff6ec7bcf5fa76d81dc4b11b132d3cdc69db3d706d57a227f"}
Sep 16 12:20:41.569297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.569234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht" event={"ID":"d18768f4-3c50-48c3-b532-0edf316c15d6","Type":"ContainerStarted","Data":"d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099"}
Sep 16 12:20:41.569297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.569286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht" event={"ID":"d18768f4-3c50-48c3-b532-0edf316c15d6","Type":"ContainerStarted","Data":"17eb79033d36c2c6a1c5ad436a1eba22f628278b49d172b31d632840b3a2d41d"}
Sep 16 12:20:41.569493 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.569318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:41.585013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.584957    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc" podStartSLOduration=1.5849385520000001 podStartE2EDuration="1.584938552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:41.582605983 +0000 UTC m=+1128.994622879" watchObservedRunningTime="2026-09-16 12:20:41.584938552 +0000 UTC m=+1128.996955449"
Sep 16 12:20:41.610586 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.604400    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht" podStartSLOduration=1.604382075 podStartE2EDuration="1.604382075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:41.603123403 +0000 UTC m=+1129.015140299" watchObservedRunningTime="2026-09-16 12:20:41.604382075 +0000 UTC m=+1129.016398971"
Sep 16 12:20:41.980196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:41.980159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:42.385081 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.385044    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 12:20:42.385509 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.385105    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 12:20:42.575346 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.575310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:42.576354 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.576334    2577 generic.go:356] "Generic (PLEG): container finished" podID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerID="0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84" exitCode=0
Sep 16 12:20:42.576441 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.576405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" event={"ID":"48dd50c1-e7b9-44be-ab2f-2ad8d8588327","Type":"ContainerDied","Data":"0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84"}
Sep 16 12:20:42.576835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.576816    2577 scope.go:117] "RemoveContainer" containerID="0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84"
Sep 16 12:20:42.577890 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.577861    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:20:42.577991 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.577950    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:20:42.578054 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:42.577996    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:20:43.181913 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:43.181884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:43.584234 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:43.584147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" event={"ID":"48dd50c1-e7b9-44be-ab2f-2ad8d8588327","Type":"ContainerStarted","Data":"a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4"}
Sep 16 12:20:44.255299 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:44.255257    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:44.255622 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:44.255597    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 12:20:44.255700 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:44.255650    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 12:20:45.255644 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:45.255604    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 12:20:45.256044 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:45.255674    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 12:20:45.582330 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:45.582248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:45.690938 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:45.690879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:45.905875 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:45.905840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:46.256153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:46.256045    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 12:20:46.256153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:46.256117    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 12:20:46.485523 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:46.485487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:46.863378 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:46.863344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:47.245908 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:47.245866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:47.255881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:47.255853    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 12:20:47.256016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:47.255902    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 12:20:48.053795 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:48.053756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:48.154636 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:48.154601    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:48.256422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:48.256378    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 16 12:20:48.256639 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:48.256442    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 16 12:20:49.260013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:49.259982    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:49.260390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:49.260351    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:49.261050 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:49.261032    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:20:49.580614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:49.580495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:49.797310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:49.797275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:49.903368 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:49.903331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:50.074384 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:50.074349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:50.379988 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:50.379932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:51.182381 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:51.182343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:51.595416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:51.595337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:20:51.597185 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:51.597156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:51.855868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:51.855784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:52.532708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:52.532677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:52.579189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:52.579157    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:20:52.579349 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:52.579228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:20:53.249688 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.249656    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"]
Sep 16 12:20:53.253079 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.253059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.255771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.255745    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-db42a6-serving-cert\""
Sep 16 12:20:53.268315 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.268290    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"]
Sep 16 12:20:53.296810 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.296784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:53.310093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.310063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.310206 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.310105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2xz\" (UniqueName: \"kubernetes.io/projected/957908ce-a038-41f4-8da0-e55af23d54e5-kube-api-access-4j2xz\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.310206 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.310128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957908ce-a038-41f4-8da0-e55af23d54e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.310344 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.310213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.357950 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.357922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:53.411137 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.411100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.411297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.411151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2xz\" (UniqueName: \"kubernetes.io/projected/957908ce-a038-41f4-8da0-e55af23d54e5-kube-api-access-4j2xz\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.411297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.411185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957908ce-a038-41f4-8da0-e55af23d54e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.411297 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.411211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.411422 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:20:53.411361    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-serving-cert: secret "maas-api-e2e-shared-a-w2-db42a6-serving-cert" not found
Sep 16 12:20:53.411460 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:20:53.411429    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls podName:957908ce-a038-41f4-8da0-e55af23d54e5 nodeName:}" failed. No retries permitted until 2026-09-16 12:20:53.911407371 +0000 UTC m=+1141.323424246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls") pod "maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" (UID: "957908ce-a038-41f4-8da0-e55af23d54e5") : secret "maas-api-e2e-shared-a-w2-db42a6-serving-cert" not found
Sep 16 12:20:53.411974 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.411948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957908ce-a038-41f4-8da0-e55af23d54e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.413543 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.413520    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.419971 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.419949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2xz\" (UniqueName: \"kubernetes.io/projected/957908ce-a038-41f4-8da0-e55af23d54e5-kube-api-access-4j2xz\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.818283 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.818248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:53.912472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.912434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:53.915513 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.915489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.917962 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.917940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:53.976046 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:53.976011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:54.164730 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:54.164695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:54.202359 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:54.202325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:54.434160 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:54.434084    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"]
Sep 16 12:20:54.437463 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:20:54.437438    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957908ce_a038_41f4_8da0_e55af23d54e5.slice/crio-4e3e79542d2b5b4fa9e8646d655664782382fee441e733cdfb60cdb6597bea22 WatchSource:0}: Error finding container 4e3e79542d2b5b4fa9e8646d655664782382fee441e733cdfb60cdb6597bea22: Status 404 returned error can't find the container with id 4e3e79542d2b5b4fa9e8646d655664782382fee441e733cdfb60cdb6597bea22
Sep 16 12:20:54.645643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:54.645604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" event={"ID":"957908ce-a038-41f4-8da0-e55af23d54e5","Type":"ContainerStarted","Data":"4e3e79542d2b5b4fa9e8646d655664782382fee441e733cdfb60cdb6597bea22"}
Sep 16 12:20:55.062178 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:55.062132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:55.343013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:55.342930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:20:55.652892 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:55.652859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" event={"ID":"957908ce-a038-41f4-8da0-e55af23d54e5","Type":"ContainerStarted","Data":"8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e"}
Sep 16 12:20:55.653247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:55.653005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:20:55.671190 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:20:55.671141    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" podStartSLOduration=2.177773646 podStartE2EDuration="2.671125522s" totalImagesPullingTime="493.351876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:55.669749356 +0000 UTC m=+1143.081766263" watchObservedRunningTime="2026-09-16 12:20:55.671125522 +0000 UTC m=+1143.083142418"
Sep 16 12:21:00.308556 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:00.308522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:01.663517 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:01.663484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:21:03.085043 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:03.084985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:03.396328 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:03.396294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:03.490923 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:03.490879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:06.684112 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:06.684076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:06.895244 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:06.895211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:06.995695 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:06.995589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:07.478365 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:07.478322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:08.206798 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:08.206763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:08.974945 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:08.974905    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:09.854865 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:09.854832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:10.492911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:10.492875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:10.563887 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:10.563849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:10.853012 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:10.852875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:10.955912 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:10.955880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:11.080331 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.080295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:11.173154 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.173097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:11.222744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.222709    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"]
Sep 16 12:21:11.225327 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.225310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.228189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.228159    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8f9e6d9e\""
Sep 16 12:21:11.228363 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.228232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8f9e6d9e-dockercfg-gjx9n\""
Sep 16 12:21:11.234975 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.234949    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"]
Sep 16 12:21:11.281375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.281333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5937-df49-445b-8d67-ccc2a8a9688a-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") " pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.281558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.281405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhbh\" (UniqueName: \"kubernetes.io/projected/875d5937-df49-445b-8d67-ccc2a8a9688a-kube-api-access-slhbh\") pod \"payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") " pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.382911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.382875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5937-df49-445b-8d67-ccc2a8a9688a-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") " pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.383091 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.382928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhbh\" (UniqueName: \"kubernetes.io/projected/875d5937-df49-445b-8d67-ccc2a8a9688a-kube-api-access-slhbh\") pod \"payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") " pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.383633 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.383615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5937-df49-445b-8d67-ccc2a8a9688a-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") " pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.393007 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.392983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhbh\" (UniqueName: \"kubernetes.io/projected/875d5937-df49-445b-8d67-ccc2a8a9688a-kube-api-access-slhbh\") pod \"payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") " pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.538265 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.538174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:11.647684 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.647652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:11.809188 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:11.809164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"]
Sep 16 12:21:11.811230 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:21:11.811201    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875d5937_df49_445b_8d67_ccc2a8a9688a.slice/crio-678b5a9b8902a036200424842738c824d6d430e18d905e914cf1f1d92af594da WatchSource:0}: Error finding container 678b5a9b8902a036200424842738c824d6d430e18d905e914cf1f1d92af594da: Status 404 returned error can't find the container with id 678b5a9b8902a036200424842738c824d6d430e18d905e914cf1f1d92af594da
Sep 16 12:21:12.383970 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:12.383937    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:12.753694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:12.753594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7" event={"ID":"875d5937-df49-445b-8d67-ccc2a8a9688a","Type":"ContainerStarted","Data":"2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f"}
Sep 16 12:21:12.753694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:12.753679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7" event={"ID":"875d5937-df49-445b-8d67-ccc2a8a9688a","Type":"ContainerStarted","Data":"678b5a9b8902a036200424842738c824d6d430e18d905e914cf1f1d92af594da"}
Sep 16 12:21:12.753942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:12.753709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:12.771177 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:12.771137    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7" podStartSLOduration=1.77112217 podStartE2EDuration="1.77112217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:12.769854776 +0000 UTC m=+1160.181871673" watchObservedRunningTime="2026-09-16 12:21:12.77112217 +0000 UTC m=+1160.183139062"
Sep 16 12:21:13.061826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:13.061743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:13.788473 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:13.788440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:14.231379 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:14.231343    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"]
Sep 16 12:21:14.264144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:14.264113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:14.484897 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:14.484808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:14.788750 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:14.788666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:21:14.901835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:14.901796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.079748 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.079654    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.280108 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.280038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.452738 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.452699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.565077 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.565041    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"]
Sep 16 12:21:15.568682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.568648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.571457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.571437    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c03953-openshift-default-dockercfg-4zwvq\""
Sep 16 12:21:15.580771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.580746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"]
Sep 16 12:21:15.592304 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.592270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.625502 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-credential-socket\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.625725 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlh4\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-kube-api-access-btlh4\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.625725 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-envoy\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.625725 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-certs\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.625725 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istiod-ca-cert\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.625953 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-data\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.625953 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-podinfo\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.625953 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-token\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.626110 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.625985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-socket\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.726841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.726741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlh4\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-kube-api-access-btlh4\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.726841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.726786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-envoy\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727078 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.726879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-certs\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727078 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.726899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istiod-ca-cert\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727078 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.726921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-data\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727078 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.726956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-podinfo\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727078 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.726985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-token\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727078 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.727048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-socket\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.727092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-credential-socket\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727669 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.727556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-socket\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727669 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.727648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-certs\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.727835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istiod-ca-cert\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.727919 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.727896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-credential-socket\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.728110 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.728090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-data\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.729490 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.729461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-envoy\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.730013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.729990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-podinfo\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.734338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.734312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlh4\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-kube-api-access-btlh4\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.734562 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.734542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-token\") pod \"e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") " pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.820931 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.820899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:15.881359 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.881320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:15.924406 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:15.924367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:16.183198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.183169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"]
Sep 16 12:21:16.184847 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:21:16.184821    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fbd44d_86b0_46d4_82e9_2ee553202e5b.slice/crio-3dbe8fae5487a905d47c7c312ec0b69b32b5f490b44c12abf30b68d54c39b987 WatchSource:0}: Error finding container 3dbe8fae5487a905d47c7c312ec0b69b32b5f490b44c12abf30b68d54c39b987: Status 404 returned error can't find the container with id 3dbe8fae5487a905d47c7c312ec0b69b32b5f490b44c12abf30b68d54c39b987
Sep 16 12:21:16.187825 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.187785    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:21:16.187926 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.187866    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:21:16.187926 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.187909    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:21:16.593411 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.593299    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:16.778181 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.777979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" event={"ID":"97fbd44d-86b0-46d4-82e9-2ee553202e5b","Type":"ContainerStarted","Data":"cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b"}
Sep 16 12:21:16.778181 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.778096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" event={"ID":"97fbd44d-86b0-46d4-82e9-2ee553202e5b","Type":"ContainerStarted","Data":"3dbe8fae5487a905d47c7c312ec0b69b32b5f490b44c12abf30b68d54c39b987"}
Sep 16 12:21:16.804878 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.804824    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podStartSLOduration=1.804808814 podStartE2EDuration="1.804808814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:16.803050067 +0000 UTC m=+1164.215066988" watchObservedRunningTime="2026-09-16 12:21:16.804808814 +0000 UTC m=+1164.216825711"
Sep 16 12:21:16.882069 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:16.882038    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:17.360577 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:17.360480    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:17.469245 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:17.469206    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"]
Sep 16 12:21:17.469495 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:17.469467    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm" podUID="319f309b-440b-4ab9-89ed-16e5de33736a" containerName="istio-proxy" containerID="cri-o://5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163" gracePeriod=30
Sep 16 12:21:17.492524 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:17.492489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:17.735006 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:17.734972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:17.874887 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:17.874857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:17.887814 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:17.887787    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:18.095025 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:18.094935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:18.789325 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:18.789292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:18.790371 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:18.790351    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:18.882393 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:18.882360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:19.015415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:19.015379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:19.689084 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:19.689049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:20.478681 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:20.478648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:21.082501 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:21.082467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:21.756340 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:21.756301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:22.186202 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.186165    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:22.458978 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.458884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:22.725888 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.725858    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:21:22.800771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.800732    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319f309b-440b-4ab9-89ed-16e5de33736a-istio-podinfo\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.800783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-certs\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.800807    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-envoy\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.800825    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnxl6\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-kube-api-access-fnxl6\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-credential-socket\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-istio-token\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801112    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-socket\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-data\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319f309b-440b-4ab9-89ed-16e5de33736a-istiod-ca-cert\") pod \"319f309b-440b-4ab9-89ed-16e5de33736a\" (UID: \"319f309b-440b-4ab9-89ed-16e5de33736a\") "
Sep 16 12:21:22.801661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:22.801661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:22.801661 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-data" (OuterVolumeSpecName: "istio-data") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:22.801869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801717    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.801869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319f309b-440b-4ab9-89ed-16e5de33736a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:22.801869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801740    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.801869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801771    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.801869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.801788    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.803108 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.803084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-kube-api-access-fnxl6" (OuterVolumeSpecName: "kube-api-access-fnxl6") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "kube-api-access-fnxl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:22.803108 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.803090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-istio-token" (OuterVolumeSpecName: "istio-token") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:22.803230 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.803140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/319f309b-440b-4ab9-89ed-16e5de33736a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:21:22.803266 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.803250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "319f309b-440b-4ab9-89ed-16e5de33736a" (UID: "319f309b-440b-4ab9-89ed-16e5de33736a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:22.812135 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.812110    2577 generic.go:356] "Generic (PLEG): container finished" podID="319f309b-440b-4ab9-89ed-16e5de33736a" containerID="5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163" exitCode=0
Sep 16 12:21:22.812254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.812163    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"
Sep 16 12:21:22.812254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.812189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm" event={"ID":"319f309b-440b-4ab9-89ed-16e5de33736a","Type":"ContainerDied","Data":"5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163"}
Sep 16 12:21:22.812254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.812243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm" event={"ID":"319f309b-440b-4ab9-89ed-16e5de33736a","Type":"ContainerDied","Data":"f5e0bc660b87569d903af32f0768ed974a65c40e4cb3acfb074547cf550b2798"}
Sep 16 12:21:22.812254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.812256    2577 scope.go:117] "RemoveContainer" containerID="5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163"
Sep 16 12:21:22.822876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.822856    2577 scope.go:117] "RemoveContainer" containerID="5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163"
Sep 16 12:21:22.823204 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:21:22.823184    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163\": container with ID starting with 5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163 not found: ID does not exist" containerID="5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163"
Sep 16 12:21:22.823263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.823215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163"} err="failed to get container status \"5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163\": rpc error: code = NotFound desc = could not find container \"5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163\": container with ID starting with 5a9a79c17bc262685f714ba3964750340908af2d497a9ba7c5a5548e69e7a163 not found: ID does not exist"
Sep 16 12:21:22.834036 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.834007    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"]
Sep 16 12:21:22.843776 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.842108    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c26b8c-openshift-default-64c779665b-7lwtm"]
Sep 16 12:21:22.897654 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.897614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:22.902844 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.902822    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319f309b-440b-4ab9-89ed-16e5de33736a-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.902935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.902869    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnxl6\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-kube-api-access-fnxl6\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.902935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.902880    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319f309b-440b-4ab9-89ed-16e5de33736a-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.902935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.902890    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319f309b-440b-4ab9-89ed-16e5de33736a-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:22.902935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:22.902898    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319f309b-440b-4ab9-89ed-16e5de33736a-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:23.060013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.059917    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"]
Sep 16 12:21:23.060408 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.060366    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319f309b-440b-4ab9-89ed-16e5de33736a" containerName="istio-proxy"
Sep 16 12:21:23.060408 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.060383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="319f309b-440b-4ab9-89ed-16e5de33736a" containerName="istio-proxy"
Sep 16 12:21:23.060626 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.060507    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="319f309b-440b-4ab9-89ed-16e5de33736a" containerName="istio-proxy"
Sep 16 12:21:23.064641 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.064616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.067126 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.067105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c03953-serving-cert\""
Sep 16 12:21:23.073816 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.073793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"]
Sep 16 12:21:23.113686 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.113654    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319f309b-440b-4ab9-89ed-16e5de33736a" path="/var/lib/kubelet/pods/319f309b-440b-4ab9-89ed-16e5de33736a/volumes"
Sep 16 12:21:23.162996 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.162966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:23.205391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.205355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.205599 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.205397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.205599 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.205508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.205743 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.205653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26n8r\" (UniqueName: \"kubernetes.io/projected/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-kube-api-access-26n8r\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.263301 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.263267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:23.306513 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.306477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26n8r\" (UniqueName: \"kubernetes.io/projected/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-kube-api-access-26n8r\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.306708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.306537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.306708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.306592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.306708 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.306644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.307303 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.307279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.309184 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.309164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.309284 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.309183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.318606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.318543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26n8r\" (UniqueName: \"kubernetes.io/projected/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-kube-api-access-26n8r\") pod \"maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.364052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.364018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:23.376309 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.376280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:23.650281 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.650258    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"]
Sep 16 12:21:23.653148 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:21:23.653116    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01e39c1_73fd_4567_b6a9_bfd5e992cb43.slice/crio-f55160e278ab9564aa03207025b7af0ee3a2af097420cd7dbf89837b7e7d1d26 WatchSource:0}: Error finding container f55160e278ab9564aa03207025b7af0ee3a2af097420cd7dbf89837b7e7d1d26: Status 404 returned error can't find the container with id f55160e278ab9564aa03207025b7af0ee3a2af097420cd7dbf89837b7e7d1d26
Sep 16 12:21:23.683444 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.683387    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:23.762000 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.761967    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:21:23.782152 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.782125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:23.821210 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:23.821178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b" event={"ID":"d01e39c1-73fd-4567-b6a9-bfd5e992cb43","Type":"ContainerStarted","Data":"f55160e278ab9564aa03207025b7af0ee3a2af097420cd7dbf89837b7e7d1d26"}
Sep 16 12:21:24.082619 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:24.082586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:24.828599 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:24.828542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b" event={"ID":"d01e39c1-73fd-4567-b6a9-bfd5e992cb43","Type":"ContainerStarted","Data":"f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422"}
Sep 16 12:21:24.829144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:24.828661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:24.846780 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:24.846731    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b" podStartSLOduration=1.4281257059999999 podStartE2EDuration="1.84671568s" totalImagesPullingTime="418.589974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:24.845277016 +0000 UTC m=+1172.257293915" watchObservedRunningTime="2026-09-16 12:21:24.84671568 +0000 UTC m=+1172.258732576"
Sep 16 12:21:24.985897 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:24.985854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:25.700257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:25.700223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:26.391432 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:26.391401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:26.841203 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:26.841110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:27.014614 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:27.014580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:27.670958 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:27.670917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:28.507995 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:28.507959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:29.176859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.176826    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:29.253659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.253614    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584" podUID="e54a176d-1a98-49b3-99ff-337f28adc0c9" containerName="main" containerID="cri-o://6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc" gracePeriod=45
Sep 16 12:21:29.495241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.495215    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:21:29.666256 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666224    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-dshm\") pod \"e54a176d-1a98-49b3-99ff-337f28adc0c9\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") "
Sep 16 12:21:29.666422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-model-cache\") pod \"e54a176d-1a98-49b3-99ff-337f28adc0c9\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") "
Sep 16 12:21:29.666422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666295    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a176d-1a98-49b3-99ff-337f28adc0c9-tls-certs\") pod \"e54a176d-1a98-49b3-99ff-337f28adc0c9\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") "
Sep 16 12:21:29.666422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-home\") pod \"e54a176d-1a98-49b3-99ff-337f28adc0c9\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") "
Sep 16 12:21:29.666422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-tmp-dir\") pod \"e54a176d-1a98-49b3-99ff-337f28adc0c9\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") "
Sep 16 12:21:29.666422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckdhc\" (UniqueName: \"kubernetes.io/projected/e54a176d-1a98-49b3-99ff-337f28adc0c9-kube-api-access-ckdhc\") pod \"e54a176d-1a98-49b3-99ff-337f28adc0c9\" (UID: \"e54a176d-1a98-49b3-99ff-337f28adc0c9\") "
Sep 16 12:21:29.666744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e54a176d-1a98-49b3-99ff-337f28adc0c9" (UID: "e54a176d-1a98-49b3-99ff-337f28adc0c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:29.666744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666632    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-home" (OuterVolumeSpecName: "home") pod "e54a176d-1a98-49b3-99ff-337f28adc0c9" (UID: "e54a176d-1a98-49b3-99ff-337f28adc0c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:29.666744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666708    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-model-cache\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.666744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.666707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e54a176d-1a98-49b3-99ff-337f28adc0c9" (UID: "e54a176d-1a98-49b3-99ff-337f28adc0c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:29.668523 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.668501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-dshm" (OuterVolumeSpecName: "dshm") pod "e54a176d-1a98-49b3-99ff-337f28adc0c9" (UID: "e54a176d-1a98-49b3-99ff-337f28adc0c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:21:29.668623 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.668598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54a176d-1a98-49b3-99ff-337f28adc0c9-kube-api-access-ckdhc" (OuterVolumeSpecName: "kube-api-access-ckdhc") pod "e54a176d-1a98-49b3-99ff-337f28adc0c9" (UID: "e54a176d-1a98-49b3-99ff-337f28adc0c9"). InnerVolumeSpecName "kube-api-access-ckdhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:29.668688 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.668643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54a176d-1a98-49b3-99ff-337f28adc0c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e54a176d-1a98-49b3-99ff-337f28adc0c9" (UID: "e54a176d-1a98-49b3-99ff-337f28adc0c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:29.767233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.767151    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdhc\" (UniqueName: \"kubernetes.io/projected/e54a176d-1a98-49b3-99ff-337f28adc0c9-kube-api-access-ckdhc\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.767233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.767184    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-dshm\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.767233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.767195    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54a176d-1a98-49b3-99ff-337f28adc0c9-tls-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.767233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.767204    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-home\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.767233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.767212    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54a176d-1a98-49b3-99ff-337f28adc0c9-tmp-dir\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.859316 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.859283    2577 generic.go:356] "Generic (PLEG): container finished" podID="e54a176d-1a98-49b3-99ff-337f28adc0c9" containerID="6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc" exitCode=0
Sep 16 12:21:29.859462 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.859332    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"
Sep 16 12:21:29.859462 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.859373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584" event={"ID":"e54a176d-1a98-49b3-99ff-337f28adc0c9","Type":"ContainerDied","Data":"6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc"}
Sep 16 12:21:29.859462 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.859419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584" event={"ID":"e54a176d-1a98-49b3-99ff-337f28adc0c9","Type":"ContainerDied","Data":"92f6a9309d25aafeae0b6493b35bb5e5eb6bb6039444a82c4253697aa44509dc"}
Sep 16 12:21:29.859462 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.859440    2577 scope.go:117] "RemoveContainer" containerID="6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc"
Sep 16 12:21:29.868584 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.868553    2577 scope.go:117] "RemoveContainer" containerID="6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc"
Sep 16 12:21:29.868869 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:21:29.868851    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc\": container with ID starting with 6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc not found: ID does not exist" containerID="6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc"
Sep 16 12:21:29.868928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.868877    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc"} err="failed to get container status \"6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc\": rpc error: code = NotFound desc = could not find container \"6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc\": container with ID starting with 6c3f1d94124dec2985345d74dd34e082e4daa6cc49e803b954f06633f61353bc not found: ID does not exist"
Sep 16 12:21:29.880312 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.880289    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"]
Sep 16 12:21:29.883898 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:29.883876    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-556077/e2e-conflict-b-067c87d4-kserve-5d5f9bd59b-6c584"]
Sep 16 12:21:30.840485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:30.840450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:21:31.112229 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:31.112152    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54a176d-1a98-49b3-99ff-337f28adc0c9" path="/var/lib/kubelet/pods/e54a176d-1a98-49b3-99ff-337f28adc0c9/volumes"
Sep 16 12:21:31.490904 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:31.490871    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:38.535627 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:38.535581    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:21:38.774651 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:38.774609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:44.050761 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:44.050718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:45.294668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:45.294633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:46.097921 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:46.097833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:46.751394 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:46.751356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:47.373268 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:47.373232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:48.086872 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.086833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:48.753104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.753004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:48.791153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.791112    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:48.791321 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.791186    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 12:21:48.973538 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.973509    2577 generic.go:356] "Generic (PLEG): container finished" podID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerID="cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b" exitCode=0
Sep 16 12:21:48.973723 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.973553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" event={"ID":"97fbd44d-86b0-46d4-82e9-2ee553202e5b","Type":"ContainerDied","Data":"cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b"}
Sep 16 12:21:48.974022 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.974001    2577 scope.go:117] "RemoveContainer" containerID="cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b"
Sep 16 12:21:48.975214 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.975185    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:21:48.975285 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.975253    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:21:48.975324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:48.975285    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236196Ki","pods":"250"}
Sep 16 12:21:49.027972 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:49.027934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:49.382106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:49.382068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:49.981365 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:49.981332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" event={"ID":"97fbd44d-86b0-46d4-82e9-2ee553202e5b","Type":"ContainerStarted","Data":"bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd"}
Sep 16 12:21:50.088463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:50.088433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:21:50.882218 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:50.882178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:50.882722 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:50.882605    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:50.882722 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:50.882653    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 12:21:51.882200 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:51.882165    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:51.882390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:51.882222    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 12:21:52.882307 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:52.882270    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:52.882489 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:52.882325    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 12:21:53.214993 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:53.214907    2577 scope.go:117] "RemoveContainer" containerID="2b3816376650950414974bd52256911498e88bc4c69c4b2dcf6c0362351e9ccb"
Sep 16 12:21:53.223328 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:53.223305    2577 scope.go:117] "RemoveContainer" containerID="98b53940084d6c72d11465bdfefd8584097c090ec4ae9efb0600938d5e3aaaa9"
Sep 16 12:21:53.244844 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:53.244813    2577 scope.go:117] "RemoveContainer" containerID="7b8fe5463fac7ab42d07919b6c6d68714de9aad9677cf25867a6ee1eea96dfa8"
Sep 16 12:21:53.258229 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:53.258202    2577 scope.go:117] "RemoveContainer" containerID="5d49d769c3810e4dfdea1c9cf3f9063b35df8a3e532b4239e77b00dc474b084e"
Sep 16 12:21:53.882269 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:53.882225    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:53.882461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:53.882294    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 12:21:54.881850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:54.881811    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 12:21:54.882267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:54.881885    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 12:21:55.886101 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:55.886060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:55.886465 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:55.886348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:55.887222 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:55.887203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:21:59.602472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:21:59.602435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:33.586158 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:22:33.586103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:22:40.999356 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:22:40.999317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:22:53.294695 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:22:53.294616    2577 scope.go:117] "RemoveContainer" containerID="26b86add5fc79d1c40db3600951f3783f0bd604cb5b9d3e4484460706143a580"
Sep 16 12:22:59.686874 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:22:59.686836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:06.589913 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:06.589875    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:23:14.655107 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.655075    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"]
Sep 16 12:23:14.655488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.655440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54a176d-1a98-49b3-99ff-337f28adc0c9" containerName="main"
Sep 16 12:23:14.655488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.655450    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54a176d-1a98-49b3-99ff-337f28adc0c9" containerName="main"
Sep 16 12:23:14.655656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.655522    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54a176d-1a98-49b3-99ff-337f28adc0c9" containerName="main"
Sep 16 12:23:14.659931 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.659910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.662511 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.662487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ff057a\"/\"distinct-2-w0-ff057a-kserve-self-signed-certs\""
Sep 16 12:23:14.668398 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.668374    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"]
Sep 16 12:23:14.681990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.681962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:14.763741 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.763696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-model-cache\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.763914 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.763748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7vk\" (UniqueName: \"kubernetes.io/projected/8284bc1e-56eb-4550-aa46-5a7261557111-kube-api-access-gt7vk\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.763914 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.763788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-tmp-dir\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.763914 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.763842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8284bc1e-56eb-4550-aa46-5a7261557111-tls-certs\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.763914 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.763858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-dshm\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.763914 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.763890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-home\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865186 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-model-cache\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865186 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7vk\" (UniqueName: \"kubernetes.io/projected/8284bc1e-56eb-4550-aa46-5a7261557111-kube-api-access-gt7vk\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-tmp-dir\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8284bc1e-56eb-4550-aa46-5a7261557111-tls-certs\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-dshm\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865446 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-home\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865683 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-model-cache\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865683 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-tmp-dir\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.865780 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.865754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-home\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.867600 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.867548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-dshm\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.867744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.867720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8284bc1e-56eb-4550-aa46-5a7261557111-tls-certs\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.872809 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.872786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7vk\" (UniqueName: \"kubernetes.io/projected/8284bc1e-56eb-4550-aa46-5a7261557111-kube-api-access-gt7vk\") pod \"distinct-2-w0-ff057a-kserve-5889579c99-z27jr\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") " pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:14.970264 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:14.970159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:15.240152 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:15.240126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"]
Sep 16 12:23:15.242727 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:23:15.242697    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8284bc1e_56eb_4550_aa46_5a7261557111.slice/crio-44368c9137c2fef5147238068db450fd81baa24b3024cee466310807978c4372 WatchSource:0}: Error finding container 44368c9137c2fef5147238068db450fd81baa24b3024cee466310807978c4372: Status 404 returned error can't find the container with id 44368c9137c2fef5147238068db450fd81baa24b3024cee466310807978c4372
Sep 16 12:23:15.438479 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:15.438439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr" event={"ID":"8284bc1e-56eb-4550-aa46-5a7261557111","Type":"ContainerStarted","Data":"b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763"}
Sep 16 12:23:15.438698 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:15.438489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr" event={"ID":"8284bc1e-56eb-4550-aa46-5a7261557111","Type":"ContainerStarted","Data":"44368c9137c2fef5147238068db450fd81baa24b3024cee466310807978c4372"}
Sep 16 12:23:15.456466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:15.456399    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr" podStartSLOduration=1.456380239 podStartE2EDuration="1.456380239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:15.454229437 +0000 UTC m=+1282.866246334" watchObservedRunningTime="2026-09-16 12:23:15.456380239 +0000 UTC m=+1282.868397134"
Sep 16 12:23:24.970995 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:24.970948    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:24.971635 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:24.971018    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:24.984081 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:24.984046    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:25.511941 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:25.511909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:23:32.046838 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:32.046805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:36.131328 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:36.131294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:23:42.010895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:42.010862    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:23:54.748121 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:23:54.748085    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:15.388772 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:15.388681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:24:21.353650 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:21.353614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:21.997505 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:21.997472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:22.970219 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:22.970181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:23.609018 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:23.608980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:24.391062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:24.391034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:25.317250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:25.317214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:26.022813 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:26.022783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:26.887892 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:26.887855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:27.581638 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:27.581606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:28.569852 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:28.569821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:36.496585 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.496536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:36.532482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.532439    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"]
Sep 16 12:24:36.532761 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.532726    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7" podUID="875d5937-df49-445b-8d67-ccc2a8a9688a" containerName="payload-processing" containerID="cri-o://2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f" gracePeriod=30
Sep 16 12:24:36.594200 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.594152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:36.693467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.693411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:36.785760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.785735    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:24:36.793839 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.793803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:36.855066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.855026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5937-df49-445b-8d67-ccc2a8a9688a-plugins-config-volume\") pod \"875d5937-df49-445b-8d67-ccc2a8a9688a\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") "
Sep 16 12:24:36.855247 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.855082    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slhbh\" (UniqueName: \"kubernetes.io/projected/875d5937-df49-445b-8d67-ccc2a8a9688a-kube-api-access-slhbh\") pod \"875d5937-df49-445b-8d67-ccc2a8a9688a\" (UID: \"875d5937-df49-445b-8d67-ccc2a8a9688a\") "
Sep 16 12:24:36.855430 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.855407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875d5937-df49-445b-8d67-ccc2a8a9688a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "875d5937-df49-445b-8d67-ccc2a8a9688a" (UID: "875d5937-df49-445b-8d67-ccc2a8a9688a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:24:36.857339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.857287    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875d5937-df49-445b-8d67-ccc2a8a9688a-kube-api-access-slhbh" (OuterVolumeSpecName: "kube-api-access-slhbh") pod "875d5937-df49-445b-8d67-ccc2a8a9688a" (UID: "875d5937-df49-445b-8d67-ccc2a8a9688a"). InnerVolumeSpecName "kube-api-access-slhbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:24:36.872085 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.872050    2577 generic.go:356] "Generic (PLEG): container finished" podID="875d5937-df49-445b-8d67-ccc2a8a9688a" containerID="2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f" exitCode=0
Sep 16 12:24:36.872228 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.872086    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"
Sep 16 12:24:36.872228 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.872122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7" event={"ID":"875d5937-df49-445b-8d67-ccc2a8a9688a","Type":"ContainerDied","Data":"2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f"}
Sep 16 12:24:36.872228 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.872179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7" event={"ID":"875d5937-df49-445b-8d67-ccc2a8a9688a","Type":"ContainerDied","Data":"678b5a9b8902a036200424842738c824d6d430e18d905e914cf1f1d92af594da"}
Sep 16 12:24:36.872228 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.872210    2577 scope.go:117] "RemoveContainer" containerID="2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f"
Sep 16 12:24:36.882813 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.882656    2577 scope.go:117] "RemoveContainer" containerID="2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f"
Sep 16 12:24:36.883204 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:24:36.883102    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f\": container with ID starting with 2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f not found: ID does not exist" containerID="2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f"
Sep 16 12:24:36.883204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.883177    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f"} err="failed to get container status \"2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f\": rpc error: code = NotFound desc = could not find container \"2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f\": container with ID starting with 2d737d61beb6cd6f5f83b04a8952868db56559139c684757bfb0820932950a4f not found: ID does not exist"
Sep 16 12:24:36.897889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.897848    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"]
Sep 16 12:24:36.901391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.901364    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f9e6d9e-5f87c9b49-474p7"]
Sep 16 12:24:36.956626 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.956556    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/875d5937-df49-445b-8d67-ccc2a8a9688a-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:24:36.956626 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:36.956616    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhbh\" (UniqueName: \"kubernetes.io/projected/875d5937-df49-445b-8d67-ccc2a8a9688a-kube-api-access-slhbh\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:24:37.083721 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:37.083639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:37.111930 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:37.111893    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="875d5937-df49-445b-8d67-ccc2a8a9688a" path="/var/lib/kubelet/pods/875d5937-df49-445b-8d67-ccc2a8a9688a/volumes"
Sep 16 12:24:37.672067 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:37.672034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:38.441968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:38.441933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:39.197277 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:39.197241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:39.793704 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:39.793670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:39.895585 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:39.895545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:40.395191 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:40.395156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:41.083855 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.083821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:41.653463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.653428    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"]
Sep 16 12:24:41.653960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.653828    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875d5937-df49-445b-8d67-ccc2a8a9688a" containerName="payload-processing"
Sep 16 12:24:41.653960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.653841    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="875d5937-df49-445b-8d67-ccc2a8a9688a" containerName="payload-processing"
Sep 16 12:24:41.653960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.653916    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="875d5937-df49-445b-8d67-ccc2a8a9688a" containerName="payload-processing"
Sep 16 12:24:41.658249 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.658224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.661011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.660990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"kube-root-ca.crt\""
Sep 16 12:24:41.662678 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.662638    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"openshift-service-ca.crt\""
Sep 16 12:24:41.662678 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.662672    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"default-dockercfg-m695g\""
Sep 16 12:24:41.662835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.662767    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c03953\"/\"auth-test-model-c03953-kserve-self-signed-certs\""
Sep 16 12:24:41.668508 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.668480    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"]
Sep 16 12:24:41.681236 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.681210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:41.699397 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.699372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-model-cache\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.699527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.699415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-home\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.699595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.699529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tmp-dir\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.699595 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.699558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hdh\" (UniqueName: \"kubernetes.io/projected/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-kube-api-access-v8hdh\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.699673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.699610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-dshm\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.699673 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.699637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tls-certs\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.782807 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.782778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:41.801030 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-home\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tmp-dir\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hdh\" (UniqueName: \"kubernetes.io/projected/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-kube-api-access-v8hdh\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-dshm\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tls-certs\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801159 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-model-cache\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801476 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801456    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-home\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801535 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tmp-dir\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.801594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.801550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-model-cache\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.803452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.803434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-dshm\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.803541 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.803523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tls-certs\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.809519 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.809501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hdh\" (UniqueName: \"kubernetes.io/projected/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-kube-api-access-v8hdh\") pod \"auth-test-model-c03953-kserve-6d695fb547-7r9l9\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") " pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:41.970130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:41.970040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:42.242519 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:42.242490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"]
Sep 16 12:24:42.244271 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:24:42.244243    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e2d12b2_b5e5_4ad8_8a00_2b4f82c3959d.slice/crio-54e869b9b1d13531368016b8146bd3ea1b282bb3a7713f70e2586f4d80ae2f1f WatchSource:0}: Error finding container 54e869b9b1d13531368016b8146bd3ea1b282bb3a7713f70e2586f4d80ae2f1f: Status 404 returned error can't find the container with id 54e869b9b1d13531368016b8146bd3ea1b282bb3a7713f70e2586f4d80ae2f1f
Sep 16 12:24:42.579996 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:42.579913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:42.789091 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:42.789054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:24:42.906616 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:42.906553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9" event={"ID":"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d","Type":"ContainerStarted","Data":"71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568"}
Sep 16 12:24:42.906616 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:42.906618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9" event={"ID":"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d","Type":"ContainerStarted","Data":"54e869b9b1d13531368016b8146bd3ea1b282bb3a7713f70e2586f4d80ae2f1f"}
Sep 16 12:24:42.926638 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:42.926538    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9" podStartSLOduration=1.9265173519999998 podStartE2EDuration="1.926517352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:42.924460757 +0000 UTC m=+1370.336477653" watchObservedRunningTime="2026-09-16 12:24:42.926517352 +0000 UTC m=+1370.338534249"
Sep 16 12:24:44.927607 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:44.927553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:24:51.970770 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:51.970731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:51.971346 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:51.970788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:51.983500 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:51.983470    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:24:52.971048 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:24:52.971015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:25:21.022954 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:21.022917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:21.753213 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:21.753178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:25.790966 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:25.790923    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:25:27.624066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:27.624030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:27.700096 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:27.700047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:28.009090 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:28.009049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:29.278853 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:29.278809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:29.379436 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:29.379402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:29.479527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:29.479484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:30.071521 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:30.071447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:30.160088 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:30.160056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:30.576445 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:30.576407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:31.552678 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:31.552645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:37.281366 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:37.281324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:37.346364 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:37.346330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:37.970930 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:37.970896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:39.216289 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:39.216244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:44.020014 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:44.019977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:44.114324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:44.114290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:44.211721 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:44.211673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:51.498154 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:51.498074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:53.334606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:53.334558    2577 scope.go:117] "RemoveContainer" containerID="370c465f1fed346f25c3bc3b1ff02af8bd0dba330e14a84ea7cf4cfb2f8fe196"
Sep 16 12:25:55.774888 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:55.774837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:25:55.954391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:25:55.954346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:00.933782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:00.933747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:03.090171 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:03.090137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:26:15.079703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:15.079664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:16.902168 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:16.902130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:17.264541 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:17.264454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:24.176997 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:24.176955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:24.277420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:24.277383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:26.178013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:26.177979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:26.882884 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:26.882848    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:27.036866 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:27.036823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:27.940408 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:27.940367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:28.775800 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:28.775761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:29.510404 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:29.510374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:30.175470 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:30.175430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:30.787984 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:30.787947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:31.457618 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.457584    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"]
Sep 16 12:26:31.461280 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.461260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.464306 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.464282    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a6f57e-serving-cert\""
Sep 16 12:26:31.475283 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.475250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"]
Sep 16 12:26:31.545261 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.545229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:31.561064 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.561022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8rs7\" (UniqueName: \"kubernetes.io/projected/afd8bf41-fb0f-466c-9e36-0b866298cd57-kube-api-access-c8rs7\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.561421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.561089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.561421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.561167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd8bf41-fb0f-466c-9e36-0b866298cd57-openshift-service-ca\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.561421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.561220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-metrics-certs\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.645679 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.645649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:31.662505 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.662470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd8bf41-fb0f-466c-9e36-0b866298cd57-openshift-service-ca\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.662702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.662547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-metrics-certs\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.662702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.662694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8rs7\" (UniqueName: \"kubernetes.io/projected/afd8bf41-fb0f-466c-9e36-0b866298cd57-kube-api-access-c8rs7\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.662835 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.662730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.662895 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:26:31.662866    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-serving-cert: secret "maas-api-e2e-api-a-a6f57e-serving-cert" not found
Sep 16 12:26:31.662951 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:26:31.662927    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls podName:afd8bf41-fb0f-466c-9e36-0b866298cd57 nodeName:}" failed. No retries permitted until 2026-09-16 12:26:32.162907494 +0000 UTC m=+1479.574924369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls") pod "maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw" (UID: "afd8bf41-fb0f-466c-9e36-0b866298cd57") : secret "maas-api-e2e-api-a-a6f57e-serving-cert" not found
Sep 16 12:26:31.663206 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.663184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd8bf41-fb0f-466c-9e36-0b866298cd57-openshift-service-ca\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.665230 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.665206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-metrics-certs\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.671502 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.671478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8rs7\" (UniqueName: \"kubernetes.io/projected/afd8bf41-fb0f-466c-9e36-0b866298cd57-kube-api-access-c8rs7\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:31.950436 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:31.950399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:32.049988 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:32.049952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:32.167348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:32.167303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:32.170170 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:32.170144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls\") pod \"maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:32.372325 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:32.372225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:32.645623 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:32.645559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"]
Sep 16 12:26:32.646966 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:26:32.646942    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd8bf41_fb0f_466c_9e36_0b866298cd57.slice/crio-33b55caefebdf9abb6ad48004df13cb211a21224948821ff90fcc18cd9b7e4f8 WatchSource:0}: Error finding container 33b55caefebdf9abb6ad48004df13cb211a21224948821ff90fcc18cd9b7e4f8: Status 404 returned error can't find the container with id 33b55caefebdf9abb6ad48004df13cb211a21224948821ff90fcc18cd9b7e4f8
Sep 16 12:26:32.648328 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:32.648310    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:26:33.499458 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:33.499374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw" event={"ID":"afd8bf41-fb0f-466c-9e36-0b866298cd57","Type":"ContainerStarted","Data":"f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939"}
Sep 16 12:26:33.499458 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:33.499429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw" event={"ID":"afd8bf41-fb0f-466c-9e36-0b866298cd57","Type":"ContainerStarted","Data":"33b55caefebdf9abb6ad48004df13cb211a21224948821ff90fcc18cd9b7e4f8"}
Sep 16 12:26:33.499882 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:33.499480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:33.517492 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:33.517442    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw" podStartSLOduration=1.920290136 podStartE2EDuration="2.517427146s" totalImagesPullingTime="597.13701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:33.515868848 +0000 UTC m=+1480.927885747" watchObservedRunningTime="2026-09-16 12:26:33.517427146 +0000 UTC m=+1480.929444043"
Sep 16 12:26:39.510004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:39.509973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:26:39.675303 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:39.675270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:43.688497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:43.688458    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:26:44.765900 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:44.765858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:52.304958 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:52.304925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:53.355760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:53.355722    2577 scope.go:117] "RemoveContainer" containerID="f8bb0691d43e07e60dc02f7a82b02a1a612faf4d91f60869c62593a5a8bc2ff3"
Sep 16 12:26:53.764990 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:53.764957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:54.490338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:54.490301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:55.101471 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:55.101437    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:55.708058 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:55.708025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:56.615528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:56.615496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:57.088436 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:57.088403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:57.327042 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:57.327012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:57.492143 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:57.492110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:57.828985 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:57.828864    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:58.282861 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:58.282829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:58.384337 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:58.384294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:26:59.149918 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:26:59.149887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:00.372536 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:00.372475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:01.072336 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:01.072305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:01.244002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:01.243956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:01.351153 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:01.351063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:02.014648 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:02.014599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:02.813314 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:02.813281    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:02.911912 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:02.911872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:03.690050 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:03.689996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:04.367942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:04.367897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:05.190594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:05.190525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:05.895244 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:05.895210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:06.602767 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:06.602740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:06.678511 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:06.678479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:06.777998 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:06.777966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:07.155100 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:07.155067    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:07.253452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:07.253421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:08.309063 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:08.309025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:08.803917 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:08.803879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:09.711200 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:09.711159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:10.355305 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:10.355266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:11.016687 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:11.016654    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:11.636988 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:11.636949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:12.697231 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:12.697198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:13.347739 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:13.347699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:13.543924 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:13.543893    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:14.300404 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:14.300352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:15.034606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:15.034539    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:15.133618 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:15.133564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:15.888438 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:15.888404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:15.989173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:15.989135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:16.478424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:16.478393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:17.372390 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:17.372336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:17.916730 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:17.916691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:18.535468 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:18.535431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:19.254867 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:19.254832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:19.870847 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:19.870788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:20.554488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:20.554452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:21.289523 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:21.289469    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:22.075151 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:22.075113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:22.767822 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:22.767792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:23.291744 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:23.291708    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:27:23.394076 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:23.394034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:24.022403 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:24.022362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.005503 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.005461    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"]
Sep 16 12:27:27.007795 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.007771    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-556077-29826015-6nbkt"]
Sep 16 12:27:27.031377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.031324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.036388 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.036360    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"]
Sep 16 12:27:27.036702 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.036675    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb" podUID="6def81ab-d812-4022-b577-6dda69637422" containerName="payload-processing" containerID="cri-o://a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b" gracePeriod=30
Sep 16 12:27:27.113130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.113096    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c13b39-379c-495b-8839-dd5ef9553140" path="/var/lib/kubelet/pods/92c13b39-379c-495b-8839-dd5ef9553140/volumes"
Sep 16 12:27:27.126020 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.125977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.205830 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.205792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.285598 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.285543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.314232 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.314192    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:27:27.402234 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.402134    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6def81ab-d812-4022-b577-6dda69637422-plugins-config-volume\") pod \"6def81ab-d812-4022-b577-6dda69637422\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") "
Sep 16 12:27:27.402406 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.402301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tn5x\" (UniqueName: \"kubernetes.io/projected/6def81ab-d812-4022-b577-6dda69637422-kube-api-access-4tn5x\") pod \"6def81ab-d812-4022-b577-6dda69637422\" (UID: \"6def81ab-d812-4022-b577-6dda69637422\") "
Sep 16 12:27:27.402540 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.402512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6def81ab-d812-4022-b577-6dda69637422-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6def81ab-d812-4022-b577-6dda69637422" (UID: "6def81ab-d812-4022-b577-6dda69637422"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:27.404403 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.404378    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6def81ab-d812-4022-b577-6dda69637422-kube-api-access-4tn5x" (OuterVolumeSpecName: "kube-api-access-4tn5x") pod "6def81ab-d812-4022-b577-6dda69637422" (UID: "6def81ab-d812-4022-b577-6dda69637422"). InnerVolumeSpecName "kube-api-access-4tn5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:27.504066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.504031    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6def81ab-d812-4022-b577-6dda69637422-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:27.504066 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.504060    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tn5x\" (UniqueName: \"kubernetes.io/projected/6def81ab-d812-4022-b577-6dda69637422-kube-api-access-4tn5x\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:27.615785 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.615694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.705907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.705872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:27.779852 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.779816    2577 generic.go:356] "Generic (PLEG): container finished" podID="6def81ab-d812-4022-b577-6dda69637422" containerID="a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b" exitCode=0
Sep 16 12:27:27.780021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.779863    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"
Sep 16 12:27:27.780021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.779871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb" event={"ID":"6def81ab-d812-4022-b577-6dda69637422","Type":"ContainerDied","Data":"a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b"}
Sep 16 12:27:27.780021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.779912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb" event={"ID":"6def81ab-d812-4022-b577-6dda69637422","Type":"ContainerDied","Data":"53441ad0c00c4187bb03fe0906e5afafc304dfa2f91d73a193eb70e050ee18a7"}
Sep 16 12:27:27.780021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.779935    2577 scope.go:117] "RemoveContainer" containerID="a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b"
Sep 16 12:27:27.789002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.788982    2577 scope.go:117] "RemoveContainer" containerID="a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b"
Sep 16 12:27:27.789320 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:27:27.789303    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b\": container with ID starting with a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b not found: ID does not exist" containerID="a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b"
Sep 16 12:27:27.789368 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.789328    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b"} err="failed to get container status \"a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b\": rpc error: code = NotFound desc = could not find container \"a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b\": container with ID starting with a88ab17b52be335c01a9ce3791acfeae7c24c207d96be70453f1a60f2b17458b not found: ID does not exist"
Sep 16 12:27:27.803472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.803440    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"]
Sep 16 12:27:27.809015 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:27.808984    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb"]
Sep 16 12:27:28.230073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:28.230025    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-556077-785c7b5cf5-qmfwb" podUID="6def81ab-d812-4022-b577-6dda69637422" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.32:9004: i/o timeout"
Sep 16 12:27:28.496839 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:28.496750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:29.112102 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:29.112065    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6def81ab-d812-4022-b577-6dda69637422" path="/var/lib/kubelet/pods/6def81ab-d812-4022-b577-6dda69637422/volumes"
Sep 16 12:27:29.141023 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:29.140986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:29.800071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:29.800039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:30.485609 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:30.485489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:31.092951 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:31.092919    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:31.780330 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:31.780296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:31.989541 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:31.989506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:32.688017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:32.687978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:33.597401 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:33.597360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:34.047485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:34.047449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:34.147028 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:34.146997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:34.268970 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:34.268938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:34.366626 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:34.366530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:34.677461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:34.677430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:35.093670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:35.093561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:36.893942 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:36.893907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:37.001173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:37.001128    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"]
Sep 16 12:27:37.030344 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:37.030308    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"]
Sep 16 12:27:37.099656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:37.099622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:37.198816 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:37.198727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:37.774741 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:37.774704    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.083131 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.083099    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"]
Sep 16 12:27:39.083622 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.083337    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw" podUID="afd8bf41-fb0f-466c-9e36-0b866298cd57" containerName="maas-api" containerID="cri-o://f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939" gracePeriod=30
Sep 16 12:27:39.127518 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.127446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.228691 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.228659    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.294339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.294312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.339533 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.339472    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:27:39.351836 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.351807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.425968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.425933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-metrics-certs\") pod \"afd8bf41-fb0f-466c-9e36-0b866298cd57\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") "
Sep 16 12:27:39.426184 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.426036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls\") pod \"afd8bf41-fb0f-466c-9e36-0b866298cd57\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") "
Sep 16 12:27:39.426184 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.426068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8rs7\" (UniqueName: \"kubernetes.io/projected/afd8bf41-fb0f-466c-9e36-0b866298cd57-kube-api-access-c8rs7\") pod \"afd8bf41-fb0f-466c-9e36-0b866298cd57\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") "
Sep 16 12:27:39.426184 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.426128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd8bf41-fb0f-466c-9e36-0b866298cd57-openshift-service-ca\") pod \"afd8bf41-fb0f-466c-9e36-0b866298cd57\" (UID: \"afd8bf41-fb0f-466c-9e36-0b866298cd57\") "
Sep 16 12:27:39.426807 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.426746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd8bf41-fb0f-466c-9e36-0b866298cd57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "afd8bf41-fb0f-466c-9e36-0b866298cd57" (UID: "afd8bf41-fb0f-466c-9e36-0b866298cd57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:39.428306 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.428264    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "afd8bf41-fb0f-466c-9e36-0b866298cd57" (UID: "afd8bf41-fb0f-466c-9e36-0b866298cd57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:39.428405 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.428319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "afd8bf41-fb0f-466c-9e36-0b866298cd57" (UID: "afd8bf41-fb0f-466c-9e36-0b866298cd57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:39.428481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.428461    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd8bf41-fb0f-466c-9e36-0b866298cd57-kube-api-access-c8rs7" (OuterVolumeSpecName: "kube-api-access-c8rs7") pod "afd8bf41-fb0f-466c-9e36-0b866298cd57" (UID: "afd8bf41-fb0f-466c-9e36-0b866298cd57"). InnerVolumeSpecName "kube-api-access-c8rs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:39.527821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.527776    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd8bf41-fb0f-466c-9e36-0b866298cd57-openshift-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:39.527821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.527806    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-metrics-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:39.527821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.527817    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd8bf41-fb0f-466c-9e36-0b866298cd57-maas-api-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:39.527821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.527825    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8rs7\" (UniqueName: \"kubernetes.io/projected/afd8bf41-fb0f-466c-9e36-0b866298cd57-kube-api-access-c8rs7\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:39.659015 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.658979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.761375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.761338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:39.846071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.846028    2577 generic.go:356] "Generic (PLEG): container finished" podID="afd8bf41-fb0f-466c-9e36-0b866298cd57" containerID="f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939" exitCode=0
Sep 16 12:27:39.846071 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.846074    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"
Sep 16 12:27:39.846339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.846156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw" event={"ID":"afd8bf41-fb0f-466c-9e36-0b866298cd57","Type":"ContainerDied","Data":"f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939"}
Sep 16 12:27:39.846339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.846213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw" event={"ID":"afd8bf41-fb0f-466c-9e36-0b866298cd57","Type":"ContainerDied","Data":"33b55caefebdf9abb6ad48004df13cb211a21224948821ff90fcc18cd9b7e4f8"}
Sep 16 12:27:39.846339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.846281    2577 scope.go:117] "RemoveContainer" containerID="f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939"
Sep 16 12:27:39.856478 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.856450    2577 scope.go:117] "RemoveContainer" containerID="f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939"
Sep 16 12:27:39.856832 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:27:39.856812    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939\": container with ID starting with f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939 not found: ID does not exist" containerID="f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939"
Sep 16 12:27:39.856916 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.856840    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939"} err="failed to get container status \"f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939\": rpc error: code = NotFound desc = could not find container \"f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939\": container with ID starting with f1a60a2675cd37435b7b12198d0b9efff22ada1656c3ad070f730fb49caf4939 not found: ID does not exist"
Sep 16 12:27:39.868828 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.868795    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"]
Sep 16 12:27:39.872478 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:39.872448    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a6f57e-7d87b758bb-2j5gw"]
Sep 16 12:27:40.605850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:40.605812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:41.112556 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:41.112522    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd8bf41-fb0f-466c-9e36-0b866298cd57" path="/var/lib/kubelet/pods/afd8bf41-fb0f-466c-9e36-0b866298cd57/volumes"
Sep 16 12:27:41.261967 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:41.261923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:42.175178 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:42.175141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:42.781067 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:42.781023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:43.426127 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:43.426093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:44.105324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:44.105287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:45.664644 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:45.664555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:45.725900 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:45.725866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:45.936130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:45.936044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:46.569476 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:46.569446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:47.507151 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:47.507112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:48.011216 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:48.011184    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:48.079558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:48.079481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:48.143774 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:48.143735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:48.234645 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:48.234607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:48.300178 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:48.300095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:48.626271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:48.626187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:49.032106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:49.032068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:49.866984 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:49.866940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:50.534944 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:50.534911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:50.648500 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:50.648463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:50.743562 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:50.743531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:51.173316 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:51.173277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:51.446538 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:51.446452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:52.020946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.020900    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr" podUID="8284bc1e-56eb-4550-aa46-5a7261557111" containerName="main" containerID="cri-o://b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763" gracePeriod=45
Sep 16 12:27:52.049771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.049710    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c" podUID="069b2328-0a36-4957-8529-cf544bc62bbf" containerName="main" containerID="cri-o://dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b" gracePeriod=45
Sep 16 12:27:52.058782 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.058750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:52.330220 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.330194    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:27:52.333233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.333208    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.346706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt7vk\" (UniqueName: \"kubernetes.io/projected/8284bc1e-56eb-4550-aa46-5a7261557111-kube-api-access-gt7vk\") pod \"8284bc1e-56eb-4550-aa46-5a7261557111\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.346789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/069b2328-0a36-4957-8529-cf544bc62bbf-tls-certs\") pod \"069b2328-0a36-4957-8529-cf544bc62bbf\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.346837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8284bc1e-56eb-4550-aa46-5a7261557111-tls-certs\") pod \"8284bc1e-56eb-4550-aa46-5a7261557111\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.346866    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-tmp-dir\") pod \"8284bc1e-56eb-4550-aa46-5a7261557111\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.346894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-447hh\" (UniqueName: \"kubernetes.io/projected/069b2328-0a36-4957-8529-cf544bc62bbf-kube-api-access-447hh\") pod \"069b2328-0a36-4957-8529-cf544bc62bbf\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.346944    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-dshm\") pod \"069b2328-0a36-4957-8529-cf544bc62bbf\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.346981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-dshm\") pod \"8284bc1e-56eb-4550-aa46-5a7261557111\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.347026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-model-cache\") pod \"8284bc1e-56eb-4550-aa46-5a7261557111\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.347057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-tmp-dir\") pod \"069b2328-0a36-4957-8529-cf544bc62bbf\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.347093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-model-cache\") pod \"069b2328-0a36-4957-8529-cf544bc62bbf\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.347136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-home\") pod \"8284bc1e-56eb-4550-aa46-5a7261557111\" (UID: \"8284bc1e-56eb-4550-aa46-5a7261557111\") "
Sep 16 12:27:52.347148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.347163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-home\") pod \"069b2328-0a36-4957-8529-cf544bc62bbf\" (UID: \"069b2328-0a36-4957-8529-cf544bc62bbf\") "
Sep 16 12:27:52.347899 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.347703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8284bc1e-56eb-4550-aa46-5a7261557111" (UID: "8284bc1e-56eb-4550-aa46-5a7261557111"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.347899 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.347728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-home" (OuterVolumeSpecName: "home") pod "069b2328-0a36-4957-8529-cf544bc62bbf" (UID: "069b2328-0a36-4957-8529-cf544bc62bbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.348972 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.348296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-model-cache" (OuterVolumeSpecName: "model-cache") pod "8284bc1e-56eb-4550-aa46-5a7261557111" (UID: "8284bc1e-56eb-4550-aa46-5a7261557111"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.348972 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.348296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "069b2328-0a36-4957-8529-cf544bc62bbf" (UID: "069b2328-0a36-4957-8529-cf544bc62bbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.348972 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.348484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "069b2328-0a36-4957-8529-cf544bc62bbf" (UID: "069b2328-0a36-4957-8529-cf544bc62bbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.349280 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.349047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-home" (OuterVolumeSpecName: "home") pod "8284bc1e-56eb-4550-aa46-5a7261557111" (UID: "8284bc1e-56eb-4550-aa46-5a7261557111"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.351346 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.351314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069b2328-0a36-4957-8529-cf544bc62bbf-kube-api-access-447hh" (OuterVolumeSpecName: "kube-api-access-447hh") pod "069b2328-0a36-4957-8529-cf544bc62bbf" (UID: "069b2328-0a36-4957-8529-cf544bc62bbf"). InnerVolumeSpecName "kube-api-access-447hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:52.351935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.351883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-dshm" (OuterVolumeSpecName: "dshm") pod "069b2328-0a36-4957-8529-cf544bc62bbf" (UID: "069b2328-0a36-4957-8529-cf544bc62bbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.351935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.351919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-dshm" (OuterVolumeSpecName: "dshm") pod "8284bc1e-56eb-4550-aa46-5a7261557111" (UID: "8284bc1e-56eb-4550-aa46-5a7261557111"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:52.352192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.352047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8284bc1e-56eb-4550-aa46-5a7261557111-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8284bc1e-56eb-4550-aa46-5a7261557111" (UID: "8284bc1e-56eb-4550-aa46-5a7261557111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:52.352192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.352165    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8284bc1e-56eb-4550-aa46-5a7261557111-kube-api-access-gt7vk" (OuterVolumeSpecName: "kube-api-access-gt7vk") pod "8284bc1e-56eb-4550-aa46-5a7261557111" (UID: "8284bc1e-56eb-4550-aa46-5a7261557111"). InnerVolumeSpecName "kube-api-access-gt7vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:52.352692 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.352654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069b2328-0a36-4957-8529-cf544bc62bbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "069b2328-0a36-4957-8529-cf544bc62bbf" (UID: "069b2328-0a36-4957-8529-cf544bc62bbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:52.448754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448712    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-model-cache\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.448754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448749    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-tmp-dir\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.448754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448760    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-model-cache\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448771    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-home\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448782    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-home\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448794    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7vk\" (UniqueName: \"kubernetes.io/projected/8284bc1e-56eb-4550-aa46-5a7261557111-kube-api-access-gt7vk\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448806    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/069b2328-0a36-4957-8529-cf544bc62bbf-tls-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448819    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8284bc1e-56eb-4550-aa46-5a7261557111-tls-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448829    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-tmp-dir\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448839    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447hh\" (UniqueName: \"kubernetes.io/projected/069b2328-0a36-4957-8529-cf544bc62bbf-kube-api-access-447hh\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448849    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/069b2328-0a36-4957-8529-cf544bc62bbf-dshm\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.449009 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.448861    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8284bc1e-56eb-4550-aa46-5a7261557111-dshm\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:27:52.740719 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.740682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:52.920040 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.920000    2577 generic.go:356] "Generic (PLEG): container finished" podID="8284bc1e-56eb-4550-aa46-5a7261557111" containerID="b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763" exitCode=0
Sep 16 12:27:52.920241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.920048    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"
Sep 16 12:27:52.920241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.920093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr" event={"ID":"8284bc1e-56eb-4550-aa46-5a7261557111","Type":"ContainerDied","Data":"b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763"}
Sep 16 12:27:52.920241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.920140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr" event={"ID":"8284bc1e-56eb-4550-aa46-5a7261557111","Type":"ContainerDied","Data":"44368c9137c2fef5147238068db450fd81baa24b3024cee466310807978c4372"}
Sep 16 12:27:52.920241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.920164    2577 scope.go:117] "RemoveContainer" containerID="b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763"
Sep 16 12:27:52.921480 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.921458    2577 generic.go:356] "Generic (PLEG): container finished" podID="069b2328-0a36-4957-8529-cf544bc62bbf" containerID="dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b" exitCode=0
Sep 16 12:27:52.921551 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.921522    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"
Sep 16 12:27:52.921620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.921541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c" event={"ID":"069b2328-0a36-4957-8529-cf544bc62bbf","Type":"ContainerDied","Data":"dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b"}
Sep 16 12:27:52.921620 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.921601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c" event={"ID":"069b2328-0a36-4957-8529-cf544bc62bbf","Type":"ContainerDied","Data":"024c3fa7795b1cebcdb2e8d7bccb736685ca334cd175f998f210dfba7eb36128"}
Sep 16 12:27:52.930930 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.930902    2577 scope.go:117] "RemoveContainer" containerID="b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763"
Sep 16 12:27:52.931328 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:27:52.931301    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763\": container with ID starting with b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763 not found: ID does not exist" containerID="b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763"
Sep 16 12:27:52.931427 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.931344    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763"} err="failed to get container status \"b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763\": rpc error: code = NotFound desc = could not find container \"b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763\": container with ID starting with b8df40b3259b4aaf851186f16bd42a7e095f993df42addc56574ac9d33afe763 not found: ID does not exist"
Sep 16 12:27:52.931427 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.931372    2577 scope.go:117] "RemoveContainer" containerID="dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b"
Sep 16 12:27:52.940198 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.940177    2577 scope.go:117] "RemoveContainer" containerID="dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b"
Sep 16 12:27:52.940540 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:27:52.940520    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b\": container with ID starting with dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b not found: ID does not exist" containerID="dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b"
Sep 16 12:27:52.940631 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.940549    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b"} err="failed to get container status \"dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b\": rpc error: code = NotFound desc = could not find container \"dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b\": container with ID starting with dfd0ee1782d3f46e2322501a1d6ff1ecc9714f71f74527aa5dc1d18776e0172b not found: ID does not exist"
Sep 16 12:27:52.946232 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.946205    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"]
Sep 16 12:27:52.949539 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.949512    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/premium-w0-ff057a-kserve-dbc64869c-6759c"]
Sep 16 12:27:52.959706 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.959661    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"]
Sep 16 12:27:52.965146 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:52.965116    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ff057a/distinct-2-w0-ff057a-kserve-5889579c99-z27jr"]
Sep 16 12:27:53.112252 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:53.112213    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="069b2328-0a36-4957-8529-cf544bc62bbf" path="/var/lib/kubelet/pods/069b2328-0a36-4957-8529-cf544bc62bbf/volumes"
Sep 16 12:27:53.112642 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:53.112627    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8284bc1e-56eb-4550-aa46-5a7261557111" path="/var/lib/kubelet/pods/8284bc1e-56eb-4550-aa46-5a7261557111/volumes"
Sep 16 12:27:53.374659 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:53.374549    2577 scope.go:117] "RemoveContainer" containerID="72486e5e84eb2a9a05098ec51f21784314f3aa9fef21594e0145d299e5529789"
Sep 16 12:27:54.334079 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:54.334014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:54.396264 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:54.396230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:54.635156 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:54.635119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:56.418935 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:56.418899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:56.855042 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:56.854906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:27:59.787748 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:27:59.787702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:00.284413 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:00.284371    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:28:01.961332 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:01.961279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:02.036830 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:02.036788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:02.101463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:02.101420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:02.224487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:02.224395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:02.576102 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:02.575996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:02.698705 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:02.698670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:02.949671 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:02.949632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:03.387148 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:03.387103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:04.256261 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:04.256201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:04.875005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:04.874969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:05.698105 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:05.698070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:06.534157 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:06.534119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:06.630415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:06.630381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:06.744024 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:06.743989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:08.394556 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:08.394520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:08.447821 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:08.447785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:08.676445 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:08.676362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:09.196239 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:09.196206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:09.957515 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:09.957478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:10.669353 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:10.669317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:11.270719 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:11.270678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:11.873726 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:11.873692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:12.587461 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:12.587427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:12.977797 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:12.977759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.477204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.477153    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"]
Sep 16 12:28:19.478005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.477944    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh" podUID="fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" containerName="maas-api" containerID="cri-o://6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed" gracePeriod=30
Sep 16 12:28:19.506116 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.506092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.577606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.577545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.679525 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.679486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.735551 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.735500    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:28:19.740947 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.740921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:19.908856 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.908827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls\") pod \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") "
Sep 16 12:28:19.909061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.908945    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-openshift-service-ca\") pod \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") "
Sep 16 12:28:19.909061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.908982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhgds\" (UniqueName: \"kubernetes.io/projected/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-kube-api-access-rhgds\") pod \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") "
Sep 16 12:28:19.909061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.909032    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-metrics-certs\") pod \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\" (UID: \"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05\") "
Sep 16 12:28:19.909612 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.909553    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" (UID: "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:19.911603 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.911551    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" (UID: "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:19.911903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.911873    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" (UID: "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:19.911903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:19.911882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-kube-api-access-rhgds" (OuterVolumeSpecName: "kube-api-access-rhgds") pod "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" (UID: "fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05"). InnerVolumeSpecName "kube-api-access-rhgds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:20.010045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.009961    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-openshift-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:20.010045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.009995    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhgds\" (UniqueName: \"kubernetes.io/projected/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-kube-api-access-rhgds\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:20.010045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.010005    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-metrics-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:20.010045 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.010013    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05-maas-api-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:20.049963 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.049934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:20.061329 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.061303    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" containerID="6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed" exitCode=0
Sep 16 12:28:20.061452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.061346    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"
Sep 16 12:28:20.061452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.061350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh" event={"ID":"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05","Type":"ContainerDied","Data":"6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed"}
Sep 16 12:28:20.061452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.061392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh" event={"ID":"fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05","Type":"ContainerDied","Data":"90f539646f70991a373bcd086a33bba6551ea811aac978751bfe6eafb22fda25"}
Sep 16 12:28:20.061452 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.061415    2577 scope.go:117] "RemoveContainer" containerID="6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed"
Sep 16 12:28:20.070770 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.070752    2577 scope.go:117] "RemoveContainer" containerID="6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed"
Sep 16 12:28:20.071023 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:28:20.071004    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed\": container with ID starting with 6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed not found: ID does not exist" containerID="6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed"
Sep 16 12:28:20.071082 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.071034    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed"} err="failed to get container status \"6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed\": rpc error: code = NotFound desc = could not find container \"6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed\": container with ID starting with 6a756b288662bdde45c1aa6bb81b7f05843b9138744aaecb1e9e03e77d0e43ed not found: ID does not exist"
Sep 16 12:28:20.084356 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.084331    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"]
Sep 16 12:28:20.088970 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.088951    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-867202-749f8755b9-nx7dh"]
Sep 16 12:28:20.166402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.166366    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:20.889202 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:20.889167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:21.111260 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:21.111226    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" path="/var/lib/kubelet/pods/fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05/volumes"
Sep 16 12:28:21.491785 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:21.491758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:22.306348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:22.306316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:24.087111 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:24.087080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:28:26.053869 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:26.053834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:26.115576 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:26.115541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:26.314546 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:26.314463    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"]
Sep 16 12:28:26.314758 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:26.314731    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy" containerID="cri-o://a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4" gracePeriod=30
Sep 16 12:28:26.339532 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:26.339502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:26.806582 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:26.806527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:27.547762 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:27.547719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:28.266998 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:28.266959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:28.874545 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:28.874509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:29.444770 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:29.444738    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:31.566415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.566383    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:28:31.714766 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714733    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-podinfo\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.714952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714797    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-socket\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.714952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlxp5\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-kube-api-access-qlxp5\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.714952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714866    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-token\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.714952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-credential-socket\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.714952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istiod-ca-cert\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.714952 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-certs\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.715267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.714985    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-data\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.715267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-envoy\") pod \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\" (UID: \"48dd50c1-e7b9-44be-ab2f-2ad8d8588327\") "
Sep 16 12:28:31.715267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:31.715267 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715238    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:31.715487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-data" (OuterVolumeSpecName: "istio-data") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:31.715487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:31.715487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715426    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.715487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715446    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.715487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715460    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.715786 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.715761    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:31.717204 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.717183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-kube-api-access-qlxp5" (OuterVolumeSpecName: "kube-api-access-qlxp5") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "kube-api-access-qlxp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:31.717275 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.717228    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:31.717403 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.717386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:28:31.717445 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.717408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-token" (OuterVolumeSpecName: "istio-token") pod "48dd50c1-e7b9-44be-ab2f-2ad8d8588327" (UID: "48dd50c1-e7b9-44be-ab2f-2ad8d8588327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:31.816862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.816827    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.816862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.816856    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlxp5\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-kube-api-access-qlxp5\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.816862 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.816867    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.817080 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.816877    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.817080 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.816886    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:31.817080 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:31.816893    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dd50c1-e7b9-44be-ab2f-2ad8d8588327-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:32.120617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.120513    2577 generic.go:356] "Generic (PLEG): container finished" podID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerID="a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4" exitCode=0
Sep 16 12:28:32.120617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.120559    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"
Sep 16 12:28:32.120617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.120601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" event={"ID":"48dd50c1-e7b9-44be-ab2f-2ad8d8588327","Type":"ContainerDied","Data":"a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4"}
Sep 16 12:28:32.120893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.120661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs" event={"ID":"48dd50c1-e7b9-44be-ab2f-2ad8d8588327","Type":"ContainerDied","Data":"89cb2adf366f3dd0a6e1e8f52d00c743b89cb9336bf0c1f368badd8901a29349"}
Sep 16 12:28:32.120893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.120694    2577 scope.go:117] "RemoveContainer" containerID="a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4"
Sep 16 12:28:32.137124 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.137101    2577 scope.go:117] "RemoveContainer" containerID="0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84"
Sep 16 12:28:32.145125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.145107    2577 scope.go:117] "RemoveContainer" containerID="a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4"
Sep 16 12:28:32.145442 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:28:32.145413    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4\": container with ID starting with a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4 not found: ID does not exist" containerID="a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4"
Sep 16 12:28:32.145555 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.145454    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4"} err="failed to get container status \"a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4\": rpc error: code = NotFound desc = could not find container \"a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4\": container with ID starting with a22e1c327f79ed324ee3dbf6f3251f00795188095d198f49e6fcb29bdde233b4 not found: ID does not exist"
Sep 16 12:28:32.145555 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.145473    2577 scope.go:117] "RemoveContainer" containerID="0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84"
Sep 16 12:28:32.145953 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:28:32.145915    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84\": container with ID starting with 0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84 not found: ID does not exist" containerID="0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84"
Sep 16 12:28:32.146051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.145947    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84"} err="failed to get container status \"0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84\": rpc error: code = NotFound desc = could not find container \"0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84\": container with ID starting with 0ad60cbb448d156e7cfcb2e801d7f872f2ed387e44df30ca0105d2d18fe7fb84 not found: ID does not exist"
Sep 16 12:28:32.148042 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.148017    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"]
Sep 16 12:28:32.151617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:32.151597    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-867202-openshift-default-86797fc998-tlbgs"]
Sep 16 12:28:33.111920 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:33.111892    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" path="/var/lib/kubelet/pods/48dd50c1-e7b9-44be-ab2f-2ad8d8588327/volumes"
Sep 16 12:28:37.073116 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.073044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:37.098223 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.098192    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"]
Sep 16 12:28:37.098422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.098398    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht" podUID="d18768f4-3c50-48c3-b532-0edf316c15d6" containerName="payload-processing" containerID="cri-o://d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099" gracePeriod=30
Sep 16 12:28:37.113391 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.113364    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"]
Sep 16 12:28:37.113562 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.113535    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc" podUID="a182a05f-ddfe-4248-a416-b84a058fe3a0" containerName="payload-pre-processing" containerID="cri-o://380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d" gracePeriod=30
Sep 16 12:28:37.163163 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.163112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:37.309774 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.309703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:37.360048 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.360023    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:28:37.372904 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.372876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:37.394981 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.394958    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:28:37.461240 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.461207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx4x9\" (UniqueName: \"kubernetes.io/projected/a182a05f-ddfe-4248-a416-b84a058fe3a0-kube-api-access-zx4x9\") pod \"a182a05f-ddfe-4248-a416-b84a058fe3a0\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") "
Sep 16 12:28:37.461424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.461293    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4p5x\" (UniqueName: \"kubernetes.io/projected/d18768f4-3c50-48c3-b532-0edf316c15d6-kube-api-access-r4p5x\") pod \"d18768f4-3c50-48c3-b532-0edf316c15d6\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") "
Sep 16 12:28:37.461424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.461318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a182a05f-ddfe-4248-a416-b84a058fe3a0-plugins-config-volume\") pod \"a182a05f-ddfe-4248-a416-b84a058fe3a0\" (UID: \"a182a05f-ddfe-4248-a416-b84a058fe3a0\") "
Sep 16 12:28:37.461424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.461351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18768f4-3c50-48c3-b532-0edf316c15d6-plugins-config-volume\") pod \"d18768f4-3c50-48c3-b532-0edf316c15d6\" (UID: \"d18768f4-3c50-48c3-b532-0edf316c15d6\") "
Sep 16 12:28:37.461789 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.461759    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18768f4-3c50-48c3-b532-0edf316c15d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d18768f4-3c50-48c3-b532-0edf316c15d6" (UID: "d18768f4-3c50-48c3-b532-0edf316c15d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:37.461789 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.461769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a182a05f-ddfe-4248-a416-b84a058fe3a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a182a05f-ddfe-4248-a416-b84a058fe3a0" (UID: "a182a05f-ddfe-4248-a416-b84a058fe3a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:37.463380 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.463358    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18768f4-3c50-48c3-b532-0edf316c15d6-kube-api-access-r4p5x" (OuterVolumeSpecName: "kube-api-access-r4p5x") pod "d18768f4-3c50-48c3-b532-0edf316c15d6" (UID: "d18768f4-3c50-48c3-b532-0edf316c15d6"). InnerVolumeSpecName "kube-api-access-r4p5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:37.463437 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.463365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a182a05f-ddfe-4248-a416-b84a058fe3a0-kube-api-access-zx4x9" (OuterVolumeSpecName: "kube-api-access-zx4x9") pod "a182a05f-ddfe-4248-a416-b84a058fe3a0" (UID: "a182a05f-ddfe-4248-a416-b84a058fe3a0"). InnerVolumeSpecName "kube-api-access-zx4x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:37.562859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.562823    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx4x9\" (UniqueName: \"kubernetes.io/projected/a182a05f-ddfe-4248-a416-b84a058fe3a0-kube-api-access-zx4x9\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.562859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.562852    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4p5x\" (UniqueName: \"kubernetes.io/projected/d18768f4-3c50-48c3-b532-0edf316c15d6-kube-api-access-r4p5x\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.562859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.562861    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a182a05f-ddfe-4248-a416-b84a058fe3a0-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.562859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.562870    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18768f4-3c50-48c3-b532-0edf316c15d6-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:37.689961 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.689928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:28:37.724346 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.724313    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:37.805337 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:37.805302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.150266 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.150232    2577 generic.go:356] "Generic (PLEG): container finished" podID="a182a05f-ddfe-4248-a416-b84a058fe3a0" containerID="380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d" exitCode=0
Sep 16 12:28:38.150749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.150292    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"
Sep 16 12:28:38.150749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.150327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc" event={"ID":"a182a05f-ddfe-4248-a416-b84a058fe3a0","Type":"ContainerDied","Data":"380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d"}
Sep 16 12:28:38.150749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.150398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc" event={"ID":"a182a05f-ddfe-4248-a416-b84a058fe3a0","Type":"ContainerDied","Data":"fc4e2f80cb02548ff6ec7bcf5fa76d81dc4b11b132d3cdc69db3d706d57a227f"}
Sep 16 12:28:38.150749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.150437    2577 scope.go:117] "RemoveContainer" containerID="380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d"
Sep 16 12:28:38.151841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.151669    2577 generic.go:356] "Generic (PLEG): container finished" podID="d18768f4-3c50-48c3-b532-0edf316c15d6" containerID="d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099" exitCode=0
Sep 16 12:28:38.151841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.151741    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"
Sep 16 12:28:38.151841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.151766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht" event={"ID":"d18768f4-3c50-48c3-b532-0edf316c15d6","Type":"ContainerDied","Data":"d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099"}
Sep 16 12:28:38.151841 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.151817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht" event={"ID":"d18768f4-3c50-48c3-b532-0edf316c15d6","Type":"ContainerDied","Data":"17eb79033d36c2c6a1c5ad436a1eba22f628278b49d172b31d632840b3a2d41d"}
Sep 16 12:28:38.161400 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.161378    2577 scope.go:117] "RemoveContainer" containerID="380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d"
Sep 16 12:28:38.161883 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:28:38.161857    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d\": container with ID starting with 380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d not found: ID does not exist" containerID="380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d"
Sep 16 12:28:38.161986 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.161890    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d"} err="failed to get container status \"380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d\": rpc error: code = NotFound desc = could not find container \"380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d\": container with ID starting with 380d2ba0ed62dd3bbe0b35c086473d9df0f62e47ce263ee5a7275272c03f0d4d not found: ID does not exist"
Sep 16 12:28:38.161986 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.161915    2577 scope.go:117] "RemoveContainer" containerID="d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099"
Sep 16 12:28:38.170563 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.170543    2577 scope.go:117] "RemoveContainer" containerID="d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099"
Sep 16 12:28:38.170849 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:28:38.170830    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099\": container with ID starting with d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099 not found: ID does not exist" containerID="d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099"
Sep 16 12:28:38.170902 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.170856    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099"} err="failed to get container status \"d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099\": rpc error: code = NotFound desc = could not find container \"d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099\": container with ID starting with d78d5310abd3387c6790fa78d08e62c22bf7ab65140c3c33757221333e89b099 not found: ID does not exist"
Sep 16 12:28:38.176149 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.176125    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"]
Sep 16 12:28:38.180133 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.180114    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6f8610-686c95798b-jffwc"]
Sep 16 12:28:38.190225 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.190204    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"]
Sep 16 12:28:38.193944 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.193922    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6f8610-855cbbc97f-8lvht"]
Sep 16 12:28:38.406772 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.406739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:38.766920 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:38.766842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:39.111529 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:39.111423    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a182a05f-ddfe-4248-a416-b84a058fe3a0" path="/var/lib/kubelet/pods/a182a05f-ddfe-4248-a416-b84a058fe3a0/volumes"
Sep 16 12:28:39.111919 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:39.111890    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d18768f4-3c50-48c3-b532-0edf316c15d6" path="/var/lib/kubelet/pods/d18768f4-3c50-48c3-b532-0edf316c15d6/volumes"
Sep 16 12:28:39.180182 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:39.180154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:39.676206 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:39.676172    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"]
Sep 16 12:28:39.694724 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:39.694698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:43.882915 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:43.882877    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:43.985494 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:43.985462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:44.149291 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:44.149204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:46.889662 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:46.889592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:28:49.794454 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:49.794417    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:28:54.695145 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:54.695088    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9" podUID="6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" containerName="main" containerID="cri-o://71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568" gracePeriod=45
Sep 16 12:28:54.948420 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:54.948356    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:28:55.014038 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.013999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8hdh\" (UniqueName: \"kubernetes.io/projected/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-kube-api-access-v8hdh\") pod \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") "
Sep 16 12:28:55.014258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014064    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tmp-dir\") pod \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") "
Sep 16 12:28:55.014258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-model-cache\") pod \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") "
Sep 16 12:28:55.014258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-dshm\") pod \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") "
Sep 16 12:28:55.014258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tls-certs\") pod \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") "
Sep 16 12:28:55.014466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-home\") pod \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\" (UID: \"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d\") "
Sep 16 12:28:55.014466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" (UID: "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:55.014466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" (UID: "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:55.014656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014538    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-home" (OuterVolumeSpecName: "home") pod "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" (UID: "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:55.014656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014558    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tmp-dir\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:55.014656 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.014604    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-model-cache\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:55.016429 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.016398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" (UID: "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:55.016429 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.016419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-kube-api-access-v8hdh" (OuterVolumeSpecName: "kube-api-access-v8hdh") pod "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" (UID: "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d"). InnerVolumeSpecName "kube-api-access-v8hdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:55.016606 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.016422    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-dshm" (OuterVolumeSpecName: "dshm") pod "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" (UID: "6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:55.115004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.114969    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-dshm\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:55.115004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.114997    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-tls-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:55.115004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.115006    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-home\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:55.115004 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.115014    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hdh\" (UniqueName: \"kubernetes.io/projected/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d-kube-api-access-v8hdh\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:28:55.236052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.235937    2577 generic.go:356] "Generic (PLEG): container finished" podID="6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" containerID="71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568" exitCode=0
Sep 16 12:28:55.236052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.235982    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"
Sep 16 12:28:55.236052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.236026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9" event={"ID":"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d","Type":"ContainerDied","Data":"71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568"}
Sep 16 12:28:55.236342 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.236076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9" event={"ID":"6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d","Type":"ContainerDied","Data":"54e869b9b1d13531368016b8146bd3ea1b282bb3a7713f70e2586f4d80ae2f1f"}
Sep 16 12:28:55.236342 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.236095    2577 scope.go:117] "RemoveContainer" containerID="71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568"
Sep 16 12:28:55.246221 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.246202    2577 scope.go:117] "RemoveContainer" containerID="71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568"
Sep 16 12:28:55.246445 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:28:55.246429    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568\": container with ID starting with 71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568 not found: ID does not exist" containerID="71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568"
Sep 16 12:28:55.246494 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.246453    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568"} err="failed to get container status \"71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568\": rpc error: code = NotFound desc = could not find container \"71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568\": container with ID starting with 71195945cc678916a3dd1d451fe093648caa166ddacbb74cf35303354a36a568 not found: ID does not exist"
Sep 16 12:28:55.253123 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.253100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"]
Sep 16 12:28:55.257227 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:55.257204    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c03953/auth-test-model-c03953-kserve-6d695fb547-7r9l9"]
Sep 16 12:28:57.111095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:28:57.111060    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" path="/var/lib/kubelet/pods/6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d/volumes"
Sep 16 12:29:07.578617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:29:07.578581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:13.996272 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:29:13.996236    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:17.488496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:29:17.488457    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:29:32.881810 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:29:32.881774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:29:45.886360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:29:45.886327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:29:58.959826 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:29:58.959787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:29:59.991281 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:29:59.991240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:01.781427 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:30:01.781392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:30:17.774427 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:30:17.774349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:30:20.705352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:30:20.705306    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:31:17.981098 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:31:17.981064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:31.401984 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:31:31.401946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:34.884629 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:31:34.884591    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:31:40.407871 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:31:40.407834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:31:45.423834 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:31:45.423745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:04.595070 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:04.595030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:32:14.388133 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:14.388100    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:32:21.317480 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:21.317443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:22.321424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:22.321381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:22.614833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:22.614737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:22.912122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:22.912088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:23.198502 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:23.198425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:24.832530 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:24.832495    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:32:25.397668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:25.397631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:26.292710 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:26.292668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:26.392285 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:26.392241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:27.249463 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:27.249401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:28.229551 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.229520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:28.367760 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.367716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:28.468195 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.468164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:28.775768 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.775696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:28.881190 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.881149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:28.921490 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921456    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"]
Sep 16 12:32:28.921811 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921798    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="069b2328-0a36-4957-8529-cf544bc62bbf" containerName="main"
Sep 16 12:32:28.921811 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921812    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="069b2328-0a36-4957-8529-cf544bc62bbf" containerName="main"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921825    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" containerName="maas-api"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921831    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" containerName="maas-api"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921839    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921845    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921853    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6def81ab-d812-4022-b577-6dda69637422" containerName="payload-processing"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921859    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6def81ab-d812-4022-b577-6dda69637422" containerName="payload-processing"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921870    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921877    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8284bc1e-56eb-4550-aa46-5a7261557111" containerName="main"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921882    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8284bc1e-56eb-4550-aa46-5a7261557111" containerName="main"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921890    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" containerName="main"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921894    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" containerName="main"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921903    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a182a05f-ddfe-4248-a416-b84a058fe3a0" containerName="payload-pre-processing"
Sep 16 12:32:28.921907 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921908    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a182a05f-ddfe-4248-a416-b84a058fe3a0" containerName="payload-pre-processing"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921914    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d18768f4-3c50-48c3-b532-0edf316c15d6" containerName="payload-processing"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921920    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d18768f4-3c50-48c3-b532-0edf316c15d6" containerName="payload-processing"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921926    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd8bf41-fb0f-466c-9e36-0b866298cd57" containerName="maas-api"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.921932    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd8bf41-fb0f-466c-9e36-0b866298cd57" containerName="maas-api"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922001    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6e5e5c-bcd7-4b2c-a5ac-8ff60a480c05" containerName="maas-api"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922008    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922013    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2d12b2-b5e5-4ad8-8a00-2b4f82c3959d" containerName="main"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922020    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd8bf41-fb0f-466c-9e36-0b866298cd57" containerName="maas-api"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922026    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a182a05f-ddfe-4248-a416-b84a058fe3a0" containerName="payload-pre-processing"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922032    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d18768f4-3c50-48c3-b532-0edf316c15d6" containerName="payload-processing"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922037    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="069b2328-0a36-4957-8529-cf544bc62bbf" containerName="main"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922045    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48dd50c1-e7b9-44be-ab2f-2ad8d8588327" containerName="istio-proxy"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922049    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6def81ab-d812-4022-b577-6dda69637422" containerName="payload-processing"
Sep 16 12:32:28.922347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.922055    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8284bc1e-56eb-4550-aa46-5a7261557111" containerName="main"
Sep 16 12:32:28.924736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.924716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:28.927357 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.927335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4f954\""
Sep 16 12:32:28.927537 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.927510    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4f954-dockercfg-kkbxk\""
Sep 16 12:32:28.934752 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:28.934724    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"]
Sep 16 12:32:29.008964 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.008930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qp6\" (UniqueName: \"kubernetes.io/projected/e8d71e92-935b-4eb6-abe3-0781b77a8c62-kube-api-access-c7qp6\") pod \"payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:29.009126 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.008988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d71e92-935b-4eb6-abe3-0781b77a8c62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:29.109597 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.109448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qp6\" (UniqueName: \"kubernetes.io/projected/e8d71e92-935b-4eb6-abe3-0781b77a8c62-kube-api-access-c7qp6\") pod \"payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:29.109597 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.109493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d71e92-935b-4eb6-abe3-0781b77a8c62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:29.110246 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.110214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d71e92-935b-4eb6-abe3-0781b77a8c62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:29.118969 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.118940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qp6\" (UniqueName: \"kubernetes.io/projected/e8d71e92-935b-4eb6-abe3-0781b77a8c62-kube-api-access-c7qp6\") pod \"payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:29.236185 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.236145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:29.506193 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:29.506162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"]
Sep 16 12:32:29.507850 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:32:29.507825    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d71e92_935b_4eb6_abe3_0781b77a8c62.slice/crio-5f6a07cdfcf663669e1cb235389a5a5102387fa441014481ad57a13daf86abbf WatchSource:0}: Error finding container 5f6a07cdfcf663669e1cb235389a5a5102387fa441014481ad57a13daf86abbf: Status 404 returned error can't find the container with id 5f6a07cdfcf663669e1cb235389a5a5102387fa441014481ad57a13daf86abbf
Sep 16 12:32:30.230867 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:30.230826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256" event={"ID":"e8d71e92-935b-4eb6-abe3-0781b77a8c62","Type":"ContainerStarted","Data":"71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d"}
Sep 16 12:32:30.231073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:30.230880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256" event={"ID":"e8d71e92-935b-4eb6-abe3-0781b77a8c62","Type":"ContainerStarted","Data":"5f6a07cdfcf663669e1cb235389a5a5102387fa441014481ad57a13daf86abbf"}
Sep 16 12:32:30.231073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:30.230941    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:30.249047 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:30.248993    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256" podStartSLOduration=2.248977495 podStartE2EDuration="2.248977495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:32:30.248294801 +0000 UTC m=+1837.660311912" watchObservedRunningTime="2026-09-16 12:32:30.248977495 +0000 UTC m=+1837.660994439"
Sep 16 12:32:37.586166 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:37.586123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:32:41.238099 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:41.238066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:32:47.913857 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:47.913821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:49.233411 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:49.233376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:51.129966 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:51.129930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:51.230756 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:51.230723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:51.566550 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:51.566467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:51.664511 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:51.664481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:51.783895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:51.783867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:51.883917 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:51.883883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:52.160591 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:52.160483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:52.471208 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:52.471108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:52.700769 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:52.700736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:53.186604 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:53.186558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:54.467002 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:54.466963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:55.075140 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:55.075104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:55.171156 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:55.171117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:55.271681 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:55.271650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:56.078338 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:56.078303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:56.804427 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:56.804396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:57.597011 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:57.596952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:57.695707 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:57.695670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:58.011344 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.011310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:58.117157 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.117121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:58.150868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.150834    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"]
Sep 16 12:32:58.154239 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.154210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.157515 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.157488    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3863e0\""
Sep 16 12:32:58.157515 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.157508    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3863e0-dockercfg-fqtj5\""
Sep 16 12:32:58.165442 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.165414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"]
Sep 16 12:32:58.181062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.181029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e960d15-7723-4b57-8312-95ac76ca17ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.181240 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.181084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvltp\" (UniqueName: \"kubernetes.io/projected/4e960d15-7723-4b57-8312-95ac76ca17ef-kube-api-access-zvltp\") pod \"payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.231519 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.231471    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:32:58.281631 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.281542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e960d15-7723-4b57-8312-95ac76ca17ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.281631 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.281618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvltp\" (UniqueName: \"kubernetes.io/projected/4e960d15-7723-4b57-8312-95ac76ca17ef-kube-api-access-zvltp\") pod \"payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.282263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.282245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e960d15-7723-4b57-8312-95ac76ca17ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.290101 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.290078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvltp\" (UniqueName: \"kubernetes.io/projected/4e960d15-7723-4b57-8312-95ac76ca17ef-kube-api-access-zvltp\") pod \"payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.467356 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.467323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:58.739770 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:58.739741    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"]
Sep 16 12:32:58.741764 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:32:58.741739    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e960d15_7723_4b57_8312_95ac76ca17ef.slice/crio-978a0f45882134a61d761092775e20dbb04f5915d70fc868283d988de6e2ba25 WatchSource:0}: Error finding container 978a0f45882134a61d761092775e20dbb04f5915d70fc868283d988de6e2ba25: Status 404 returned error can't find the container with id 978a0f45882134a61d761092775e20dbb04f5915d70fc868283d988de6e2ba25
Sep 16 12:32:59.381122 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:59.381079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7" event={"ID":"4e960d15-7723-4b57-8312-95ac76ca17ef","Type":"ContainerStarted","Data":"609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5"}
Sep 16 12:32:59.381335 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:59.381140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7" event={"ID":"4e960d15-7723-4b57-8312-95ac76ca17ef","Type":"ContainerStarted","Data":"978a0f45882134a61d761092775e20dbb04f5915d70fc868283d988de6e2ba25"}
Sep 16 12:32:59.381335 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:59.381191    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:32:59.400086 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:32:59.399978    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7" podStartSLOduration=1.399957254 podStartE2EDuration="1.399957254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:32:59.398652646 +0000 UTC m=+1866.810669541" watchObservedRunningTime="2026-09-16 12:32:59.399957254 +0000 UTC m=+1866.811974151"
Sep 16 12:33:01.791260 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:01.791221    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:33:10.388130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:10.388097    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:33:16.902318 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:16.902229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:19.368040 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:19.368001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:19.975967 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:19.975927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:20.076093 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:20.076052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:20.611873 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:20.611836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:21.591876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:21.591826    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:22.390346 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:22.390314    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:25.352687 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:25.352654    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7_4e960d15-7723-4b57-8312-95ac76ca17ef/payload-processing/0.log"
Sep 16 12:33:42.991270 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:42.991235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:33:47.000104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:33:47.000059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:34:05.196943 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:05.196905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:34:11.508677 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:11.508640    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256_e8d71e92-935b-4eb6-abe3-0781b77a8c62/payload-processing/0.log"
Sep 16 12:34:11.769039 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:11.768956    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7_4e960d15-7723-4b57-8312-95ac76ca17ef/payload-processing/0.log"
Sep 16 12:34:12.073341 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:12.073260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:12.447704 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:12.447662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:13.072867 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:13.072810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:15.101766 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:15.101720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:15.897481 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:15.897440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:15.991120 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:15.991090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:16.905583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:16.905548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:17.905449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:17.905403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:18.876125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:18.876092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:19.223893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:19.223847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:19.644881 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:19.644849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:19.746175 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:19.746143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:19.852809 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:19.852778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:20.075425 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.075316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:20.175997 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.175967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:20.190419 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.190385    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"]
Sep 16 12:34:20.193891 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.193875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.196847 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.196822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-555bf4-dockercfg-mgrpd\""
Sep 16 12:34:20.196847 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.196842    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-555bf4\""
Sep 16 12:34:20.204413 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.204384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"]
Sep 16 12:34:20.225175 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.225146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"]
Sep 16 12:34:20.228335 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.228316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.237761 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.237737    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"]
Sep 16 12:34:20.252377 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.252323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.252527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.252388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksrck\" (UniqueName: \"kubernetes.io/projected/1b83d408-a93d-48b4-bd86-278049f0c475-kube-api-access-ksrck\") pod \"payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.252527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.252424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83d408-a93d-48b4-bd86-278049f0c475-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.252527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.252456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkpqr\" (UniqueName: \"kubernetes.io/projected/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-kube-api-access-bkpqr\") pod \"payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.353125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.353025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.353125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.353071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksrck\" (UniqueName: \"kubernetes.io/projected/1b83d408-a93d-48b4-bd86-278049f0c475-kube-api-access-ksrck\") pod \"payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.353125 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.353100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83d408-a93d-48b4-bd86-278049f0c475-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.353414 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.353129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkpqr\" (UniqueName: \"kubernetes.io/projected/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-kube-api-access-bkpqr\") pod \"payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.353815 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.353793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.353878 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.353860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83d408-a93d-48b4-bd86-278049f0c475-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.360976 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.360941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkpqr\" (UniqueName: \"kubernetes.io/projected/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-kube-api-access-bkpqr\") pod \"payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.361393 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.361371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksrck\" (UniqueName: \"kubernetes.io/projected/1b83d408-a93d-48b4-bd86-278049f0c475-kube-api-access-ksrck\") pod \"payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.505230 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.505187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:20.539163 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.539130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:20.779968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.779943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"]
Sep 16 12:34:20.781636 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:34:20.781611    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdc7512_1ec8_4d2a_b6a7_ba1a99cc8754.slice/crio-28d540c123291e429916f7913f7ec874f7d4db078e6d5d4e2018f052149ce47c WatchSource:0}: Error finding container 28d540c123291e429916f7913f7ec874f7d4db078e6d5d4e2018f052149ce47c: Status 404 returned error can't find the container with id 28d540c123291e429916f7913f7ec874f7d4db078e6d5d4e2018f052149ce47c
Sep 16 12:34:20.819736 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:20.819700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"]
Sep 16 12:34:20.821760 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:34:20.821731    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b83d408_a93d_48b4_bd86_278049f0c475.slice/crio-de580c1b43ad7e90919b8921087b2b4e614db5b6ee2304e892d0ff256ad6951f WatchSource:0}: Error finding container de580c1b43ad7e90919b8921087b2b4e614db5b6ee2304e892d0ff256ad6951f: Status 404 returned error can't find the container with id de580c1b43ad7e90919b8921087b2b4e614db5b6ee2304e892d0ff256ad6951f
Sep 16 12:34:21.780893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.780846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc" event={"ID":"1b83d408-a93d-48b4-bd86-278049f0c475","Type":"ContainerStarted","Data":"ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8"}
Sep 16 12:34:21.781332 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.780911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc" event={"ID":"1b83d408-a93d-48b4-bd86-278049f0c475","Type":"ContainerStarted","Data":"de580c1b43ad7e90919b8921087b2b4e614db5b6ee2304e892d0ff256ad6951f"}
Sep 16 12:34:21.781332 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.780943    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:21.782468 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.782443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl" event={"ID":"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754","Type":"ContainerStarted","Data":"f217bd812b16e86c5f13732bdbe949cd9885c263621e4c85e116660700e2390d"}
Sep 16 12:34:21.782589 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.782484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl" event={"ID":"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754","Type":"ContainerStarted","Data":"28d540c123291e429916f7913f7ec874f7d4db078e6d5d4e2018f052149ce47c"}
Sep 16 12:34:21.782640 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.782597    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:21.800967 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.800915    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc" podStartSLOduration=1.800899601 podStartE2EDuration="1.800899601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:21.798013211 +0000 UTC m=+1949.210030108" watchObservedRunningTime="2026-09-16 12:34:21.800899601 +0000 UTC m=+1949.212916496"
Sep 16 12:34:21.815067 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:21.815017    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl" podStartSLOduration=1.814999802 podStartE2EDuration="1.814999802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:21.813508861 +0000 UTC m=+1949.225525760" watchObservedRunningTime="2026-09-16 12:34:21.814999802 +0000 UTC m=+1949.227016697"
Sep 16 12:34:25.191746 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:25.191701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:34:32.789592 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:32.789537    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:32.789976 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:32.789726    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:41.565511 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:41.565422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:42.383609 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:42.383555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:43.091674 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:43.091634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:43.794892 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:43.794860    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:44.399838 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:44.399810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:44.994631 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:44.994594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:45.977106 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:45.977067    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:47.397641 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:47.397601    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:48.185213 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:48.185178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:49.233430 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:49.233397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:49.678100 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:49.678065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:50.296044 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:50.295999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:50.676281 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:50.676250    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:50.776667 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:50.776628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:50.997682 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:50.997588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:57.653859 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.653823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:57.671594 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.671547    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"]
Sep 16 12:34:57.671815 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.671790    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc" podUID="1b83d408-a93d-48b4-bd86-278049f0c475" containerName="payload-processing" containerID="cri-o://ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8" gracePeriod=30
Sep 16 12:34:57.677946 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.677913    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"]
Sep 16 12:34:57.678167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.678142    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl" podUID="9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" containerName="payload-pre-processing" containerID="cri-o://f217bd812b16e86c5f13732bdbe949cd9885c263621e4c85e116660700e2390d" gracePeriod=30
Sep 16 12:34:57.711530 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.711497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:57.804165 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.804121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:57.897294 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.897168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:57.936233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.936203    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:57.959561 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.959530    2577 generic.go:356] "Generic (PLEG): container finished" podID="1b83d408-a93d-48b4-bd86-278049f0c475" containerID="ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8" exitCode=0
Sep 16 12:34:57.959754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.959604    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"
Sep 16 12:34:57.959754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.959618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc" event={"ID":"1b83d408-a93d-48b4-bd86-278049f0c475","Type":"ContainerDied","Data":"ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8"}
Sep 16 12:34:57.959754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.959678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc" event={"ID":"1b83d408-a93d-48b4-bd86-278049f0c475","Type":"ContainerDied","Data":"de580c1b43ad7e90919b8921087b2b4e614db5b6ee2304e892d0ff256ad6951f"}
Sep 16 12:34:57.959754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.959718    2577 scope.go:117] "RemoveContainer" containerID="ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8"
Sep 16 12:34:57.961189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.961164    2577 generic.go:356] "Generic (PLEG): container finished" podID="9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" containerID="f217bd812b16e86c5f13732bdbe949cd9885c263621e4c85e116660700e2390d" exitCode=0
Sep 16 12:34:57.961301 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.961233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl" event={"ID":"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754","Type":"ContainerDied","Data":"f217bd812b16e86c5f13732bdbe949cd9885c263621e4c85e116660700e2390d"}
Sep 16 12:34:57.969324 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.969271    2577 scope.go:117] "RemoveContainer" containerID="ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8"
Sep 16 12:34:57.969621 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:34:57.969591    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8\": container with ID starting with ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8 not found: ID does not exist" containerID="ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8"
Sep 16 12:34:57.969751 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.969633    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8"} err="failed to get container status \"ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8\": rpc error: code = NotFound desc = could not find container \"ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8\": container with ID starting with ae59710e2d77968dbc670f43784072a885769ad1a150505817e2e929425439e8 not found: ID does not exist"
Sep 16 12:34:57.972932 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:57.972911    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:58.007843 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.007799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksrck\" (UniqueName: \"kubernetes.io/projected/1b83d408-a93d-48b4-bd86-278049f0c475-kube-api-access-ksrck\") pod \"1b83d408-a93d-48b4-bd86-278049f0c475\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") "
Sep 16 12:34:58.008051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.007915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83d408-a93d-48b4-bd86-278049f0c475-plugins-config-volume\") pod \"1b83d408-a93d-48b4-bd86-278049f0c475\" (UID: \"1b83d408-a93d-48b4-bd86-278049f0c475\") "
Sep 16 12:34:58.008051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.007957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-plugins-config-volume\") pod \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") "
Sep 16 12:34:58.008051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.008009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkpqr\" (UniqueName: \"kubernetes.io/projected/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-kube-api-access-bkpqr\") pod \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\" (UID: \"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754\") "
Sep 16 12:34:58.008388 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.008355    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" (UID: "9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:34:58.008466 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.008361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b83d408-a93d-48b4-bd86-278049f0c475-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b83d408-a93d-48b4-bd86-278049f0c475" (UID: "1b83d408-a93d-48b4-bd86-278049f0c475"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:34:58.010258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.010230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-kube-api-access-bkpqr" (OuterVolumeSpecName: "kube-api-access-bkpqr") pod "9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" (UID: "9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754"). InnerVolumeSpecName "kube-api-access-bkpqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:58.010258 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.010238    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b83d408-a93d-48b4-bd86-278049f0c475-kube-api-access-ksrck" (OuterVolumeSpecName: "kube-api-access-ksrck") pod "1b83d408-a93d-48b4-bd86-278049f0c475" (UID: "1b83d408-a93d-48b4-bd86-278049f0c475"). InnerVolumeSpecName "kube-api-access-ksrck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:58.109757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.109696    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:34:58.109757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.109747    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkpqr\" (UniqueName: \"kubernetes.io/projected/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754-kube-api-access-bkpqr\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:34:58.109757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.109757    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksrck\" (UniqueName: \"kubernetes.io/projected/1b83d408-a93d-48b4-bd86-278049f0c475-kube-api-access-ksrck\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:34:58.109757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.109766    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b83d408-a93d-48b4-bd86-278049f0c475-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:34:58.233612 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.233511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:58.281955 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.281918    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"]
Sep 16 12:34:58.288233 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.288201    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-555bf4-6bcfbfb5d5-hpnrc"]
Sep 16 12:34:58.290443 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.290408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:58.879348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.879316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:34:58.967692 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.967611    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"
Sep 16 12:34:58.967692 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.967675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl" event={"ID":"9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754","Type":"ContainerDied","Data":"28d540c123291e429916f7913f7ec874f7d4db078e6d5d4e2018f052149ce47c"}
Sep 16 12:34:58.967939 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:58.967753    2577 scope.go:117] "RemoveContainer" containerID="f217bd812b16e86c5f13732bdbe949cd9885c263621e4c85e116660700e2390d"
Sep 16 12:34:59.032889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:59.032851    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"]
Sep 16 12:34:59.054597 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:59.054549    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-555bf4-9df584fcd-4wrsl"]
Sep 16 12:34:59.111740 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:59.111692    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b83d408-a93d-48b4-bd86-278049f0c475" path="/var/lib/kubelet/pods/1b83d408-a93d-48b4-bd86-278049f0c475/volumes"
Sep 16 12:34:59.112169 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:59.112148    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" path="/var/lib/kubelet/pods/9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754/volumes"
Sep 16 12:34:59.603060 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:34:59.603011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:00.254008 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:00.253973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:00.858257 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:00.858219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:01.989764 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:01.989728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:35:04.212754 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:04.212700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:04.323544 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:04.323497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:04.542553 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:04.542456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:08.807330 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:08.807300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:09.680927 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:09.680884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:10.592369 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:10.592333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:11.525230 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:11.525202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:12.193585 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:12.193531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:12.968992 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:12.968954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:19.122924 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:19.122884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:19.222474 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:19.222439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:21.991833 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:21.991790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:35:24.292341 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.292302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:24.317424 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.317390    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"]
Sep 16 12:35:24.317710 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.317681    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256" podUID="e8d71e92-935b-4eb6-abe3-0781b77a8c62" containerName="payload-processing" containerID="cri-o://71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d" gracePeriod=30
Sep 16 12:35:24.360844 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.360812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:24.461359 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.461326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:24.517261 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.517228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:24.577494 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.577470    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:35:24.756966 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.756932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d71e92-935b-4eb6-abe3-0781b77a8c62-plugins-config-volume\") pod \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") "
Sep 16 12:35:24.757156 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.756998    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7qp6\" (UniqueName: \"kubernetes.io/projected/e8d71e92-935b-4eb6-abe3-0781b77a8c62-kube-api-access-c7qp6\") pod \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\" (UID: \"e8d71e92-935b-4eb6-abe3-0781b77a8c62\") "
Sep 16 12:35:24.757284 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.757260    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d71e92-935b-4eb6-abe3-0781b77a8c62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8d71e92-935b-4eb6-abe3-0781b77a8c62" (UID: "e8d71e92-935b-4eb6-abe3-0781b77a8c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:35:24.759075 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.759051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d71e92-935b-4eb6-abe3-0781b77a8c62-kube-api-access-c7qp6" (OuterVolumeSpecName: "kube-api-access-c7qp6") pod "e8d71e92-935b-4eb6-abe3-0781b77a8c62" (UID: "e8d71e92-935b-4eb6-abe3-0781b77a8c62"). InnerVolumeSpecName "kube-api-access-c7qp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:24.858737 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.858665    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d71e92-935b-4eb6-abe3-0781b77a8c62-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:35:24.858737 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.858690    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qp6\" (UniqueName: \"kubernetes.io/projected/e8d71e92-935b-4eb6-abe3-0781b77a8c62-kube-api-access-c7qp6\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:35:24.889759 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.889732    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:24.957335 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:24.957301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:25.104772 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.104738    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8d71e92-935b-4eb6-abe3-0781b77a8c62" containerID="71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d" exitCode=0
Sep 16 12:35:25.104772 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.104775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256" event={"ID":"e8d71e92-935b-4eb6-abe3-0781b77a8c62","Type":"ContainerDied","Data":"71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d"}
Sep 16 12:35:25.104983 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.104784    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"
Sep 16 12:35:25.104983 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.104818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256" event={"ID":"e8d71e92-935b-4eb6-abe3-0781b77a8c62","Type":"ContainerDied","Data":"5f6a07cdfcf663669e1cb235389a5a5102387fa441014481ad57a13daf86abbf"}
Sep 16 12:35:25.104983 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.104840    2577 scope.go:117] "RemoveContainer" containerID="71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d"
Sep 16 12:35:25.114196 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.114077    2577 scope.go:117] "RemoveContainer" containerID="71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d"
Sep 16 12:35:25.114392 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:35:25.114369    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d\": container with ID starting with 71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d not found: ID does not exist" containerID="71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d"
Sep 16 12:35:25.114475 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.114407    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d"} err="failed to get container status \"71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d\": rpc error: code = NotFound desc = could not find container \"71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d\": container with ID starting with 71aae8dbd85c9e9c3d1a76c36ba434e6aedabbc9161f840eb39598652c0bbf9d not found: ID does not exist"
Sep 16 12:35:25.128506 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.128477    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"]
Sep 16 12:35:25.132421 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.132396    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f954-555f86d6cf-dv256"]
Sep 16 12:35:25.526791 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:25.526715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:26.275491 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:26.275455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:26.911339 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:26.911305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:27.111868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:27.111825    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d71e92-935b-4eb6-abe3-0781b77a8c62" path="/var/lib/kubelet/pods/e8d71e92-935b-4eb6-abe3-0781b77a8c62/volumes"
Sep 16 12:35:29.917213 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:29.917176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:35:35.189075 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:35:35.189032    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:36:06.027781 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:06.027749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:06.905647 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:06.905617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:12.381304 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:12.381267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:12.445694 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:12.445660    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:12.671056 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:12.670968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:13.966271 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:13.966233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:14.674162 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:14.674125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:15.356757 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:15.356669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:15.948600 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:15.948552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:18.692314 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:18.692276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:23.249526 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:23.249489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:23.788347 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:23.788313    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"]
Sep 16 12:36:23.788583 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:23.788542    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7" podUID="4e960d15-7723-4b57-8312-95ac76ca17ef" containerName="payload-processing" containerID="cri-o://609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5" gracePeriod=30
Sep 16 12:36:23.792368 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:23.792297    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:23.889911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:23.889879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:23.948312 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:23.948278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:24.036329 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.036300    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:36:24.088973 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.088891    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvltp\" (UniqueName: \"kubernetes.io/projected/4e960d15-7723-4b57-8312-95ac76ca17ef-kube-api-access-zvltp\") pod \"4e960d15-7723-4b57-8312-95ac76ca17ef\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") "
Sep 16 12:36:24.089119 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.088970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e960d15-7723-4b57-8312-95ac76ca17ef-plugins-config-volume\") pod \"4e960d15-7723-4b57-8312-95ac76ca17ef\" (UID: \"4e960d15-7723-4b57-8312-95ac76ca17ef\") "
Sep 16 12:36:24.089375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.089351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e960d15-7723-4b57-8312-95ac76ca17ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e960d15-7723-4b57-8312-95ac76ca17ef" (UID: "4e960d15-7723-4b57-8312-95ac76ca17ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:36:24.091096 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.091075    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e960d15-7723-4b57-8312-95ac76ca17ef-kube-api-access-zvltp" (OuterVolumeSpecName: "kube-api-access-zvltp") pod "4e960d15-7723-4b57-8312-95ac76ca17ef" (UID: "4e960d15-7723-4b57-8312-95ac76ca17ef"). InnerVolumeSpecName "kube-api-access-zvltp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:36:24.190303 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.190264    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvltp\" (UniqueName: \"kubernetes.io/projected/4e960d15-7723-4b57-8312-95ac76ca17ef-kube-api-access-zvltp\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:36:24.190303 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.190294    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e960d15-7723-4b57-8312-95ac76ca17ef-plugins-config-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:36:24.358601 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.358493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:24.386075 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.386043    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e960d15-7723-4b57-8312-95ac76ca17ef" containerID="609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5" exitCode=0
Sep 16 12:36:24.386242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.386087    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"
Sep 16 12:36:24.386242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.386115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7" event={"ID":"4e960d15-7723-4b57-8312-95ac76ca17ef","Type":"ContainerDied","Data":"609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5"}
Sep 16 12:36:24.386242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.386156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7" event={"ID":"4e960d15-7723-4b57-8312-95ac76ca17ef","Type":"ContainerDied","Data":"978a0f45882134a61d761092775e20dbb04f5915d70fc868283d988de6e2ba25"}
Sep 16 12:36:24.386242 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.386178    2577 scope.go:117] "RemoveContainer" containerID="609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5"
Sep 16 12:36:24.395076 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.395057    2577 scope.go:117] "RemoveContainer" containerID="609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5"
Sep 16 12:36:24.395340 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:36:24.395322    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5\": container with ID starting with 609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5 not found: ID does not exist" containerID="609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5"
Sep 16 12:36:24.395380 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.395348    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5"} err="failed to get container status \"609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5\": rpc error: code = NotFound desc = could not find container \"609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5\": container with ID starting with 609b8647caf9feaf1810e5db9a604e797ba1be77c738d5b2e490bf097edb83e5 not found: ID does not exist"
Sep 16 12:36:24.407928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.407894    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"]
Sep 16 12:36:24.419532 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.419311    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3863e0-8579dd499c-m8pk7"]
Sep 16 12:36:24.423088 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:24.423054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:25.112144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:25.112110    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e960d15-7723-4b57-8312-95ac76ca17ef" path="/var/lib/kubelet/pods/4e960d15-7723-4b57-8312-95ac76ca17ef/volumes"
Sep 16 12:36:25.151896 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:25.151861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:25.787929 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:25.787897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:27.387630 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:27.387594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:36:30.272926 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:30.272894    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:30.356438 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:30.356398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:30.630625 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:30.630591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:31.348094 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:31.348062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:32.089535 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:32.089496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:32.696361 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:32.696331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:33.504962 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:33.504933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:41.070999 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.070964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:41.559858 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.559819    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"]
Sep 16 12:36:41.560087 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.560046    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" podUID="957908ce-a038-41f4-8da0-e55af23d54e5" containerName="maas-api" containerID="cri-o://8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e" gracePeriod=30
Sep 16 12:36:41.587178 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.587106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:41.653095 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.653054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:41.658375 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.658339    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" podUID="957908ce-a038-41f4-8da0-e55af23d54e5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.68:8443/health\": dial tcp 10.133.0.68:8443: connect: connection refused"
Sep 16 12:36:41.707402 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.707360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:41.768237 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.768206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:41.805334 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.805308    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:36:41.958016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.957980    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j2xz\" (UniqueName: \"kubernetes.io/projected/957908ce-a038-41f4-8da0-e55af23d54e5-kube-api-access-4j2xz\") pod \"957908ce-a038-41f4-8da0-e55af23d54e5\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") "
Sep 16 12:36:41.958218 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.958027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls\") pod \"957908ce-a038-41f4-8da0-e55af23d54e5\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") "
Sep 16 12:36:41.958218 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.958059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-metrics-certs\") pod \"957908ce-a038-41f4-8da0-e55af23d54e5\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") "
Sep 16 12:36:41.958218 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.958087    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957908ce-a038-41f4-8da0-e55af23d54e5-openshift-service-ca\") pod \"957908ce-a038-41f4-8da0-e55af23d54e5\" (UID: \"957908ce-a038-41f4-8da0-e55af23d54e5\") "
Sep 16 12:36:41.958472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.958448    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957908ce-a038-41f4-8da0-e55af23d54e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "957908ce-a038-41f4-8da0-e55af23d54e5" (UID: "957908ce-a038-41f4-8da0-e55af23d54e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:36:41.960057 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.960027    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957908ce-a038-41f4-8da0-e55af23d54e5-kube-api-access-4j2xz" (OuterVolumeSpecName: "kube-api-access-4j2xz") pod "957908ce-a038-41f4-8da0-e55af23d54e5" (UID: "957908ce-a038-41f4-8da0-e55af23d54e5"). InnerVolumeSpecName "kube-api-access-4j2xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:36:41.960167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.960103    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "957908ce-a038-41f4-8da0-e55af23d54e5" (UID: "957908ce-a038-41f4-8da0-e55af23d54e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:36:41.960167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:41.960118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "957908ce-a038-41f4-8da0-e55af23d54e5" (UID: "957908ce-a038-41f4-8da0-e55af23d54e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:36:42.058925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.058883    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j2xz\" (UniqueName: \"kubernetes.io/projected/957908ce-a038-41f4-8da0-e55af23d54e5-kube-api-access-4j2xz\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:36:42.058925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.058919    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-maas-api-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:36:42.058925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.058931    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957908ce-a038-41f4-8da0-e55af23d54e5-metrics-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:36:42.058925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.058938    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957908ce-a038-41f4-8da0-e55af23d54e5-openshift-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:36:42.167100 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.167065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:42.268590 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.268486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:42.475850 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.475815    2577 generic.go:356] "Generic (PLEG): container finished" podID="957908ce-a038-41f4-8da0-e55af23d54e5" containerID="8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e" exitCode=0
Sep 16 12:36:42.476017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.475858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" event={"ID":"957908ce-a038-41f4-8da0-e55af23d54e5","Type":"ContainerDied","Data":"8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e"}
Sep 16 12:36:42.476017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.475867    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"
Sep 16 12:36:42.476017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.475901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4" event={"ID":"957908ce-a038-41f4-8da0-e55af23d54e5","Type":"ContainerDied","Data":"4e3e79542d2b5b4fa9e8646d655664782382fee441e733cdfb60cdb6597bea22"}
Sep 16 12:36:42.476017 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.475923    2577 scope.go:117] "RemoveContainer" containerID="8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e"
Sep 16 12:36:42.485824 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.485802    2577 scope.go:117] "RemoveContainer" containerID="8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e"
Sep 16 12:36:42.486159 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:36:42.486140    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e\": container with ID starting with 8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e not found: ID does not exist" containerID="8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e"
Sep 16 12:36:42.486210 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.486169    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e"} err="failed to get container status \"8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e\": rpc error: code = NotFound desc = could not find container \"8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e\": container with ID starting with 8f42426f3061223957cb232f9b2b4ab28a7ce34d8c6552782023fddfd1bfac5e not found: ID does not exist"
Sep 16 12:36:42.497975 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.497942    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"]
Sep 16 12:36:42.501759 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.501728    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-db42a6-858d56c8c6-99wl4"]
Sep 16 12:36:42.868857 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:42.868814    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:43.111839 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:43.111805    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957908ce-a038-41f4-8da0-e55af23d54e5" path="/var/lib/kubelet/pods/957908ce-a038-41f4-8da0-e55af23d54e5/volumes"
Sep 16 12:36:44.593487 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:44.593452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:36:46.042367 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:46.042303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:48.401450 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:48.401411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:48.501362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:48.501328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:48.781425 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:48.781343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:49.895005 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:49.894953    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:50.478753 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:50.478701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:59.647118 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:59.647080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"]
Sep 16 12:36:59.647670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:59.647412    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b" podUID="d01e39c1-73fd-4567-b6a9-bfd5e992cb43" containerName="maas-api" containerID="cri-o://f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422" gracePeriod=30
Sep 16 12:36:59.678423 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:59.678390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:59.752348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:59.752295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:59.819670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:59.819638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:36:59.895144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:59.895120    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:36:59.919911 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:36:59.919840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:00.028905 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.028870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-maas-api-tls\") pod \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") "
Sep 16 12:37:00.028905 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.028916    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26n8r\" (UniqueName: \"kubernetes.io/projected/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-kube-api-access-26n8r\") pod \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") "
Sep 16 12:37:00.029173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.028959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-openshift-service-ca\") pod \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") "
Sep 16 12:37:00.029173 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.028997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-metrics-certs\") pod \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\" (UID: \"d01e39c1-73fd-4567-b6a9-bfd5e992cb43\") "
Sep 16 12:37:00.029784 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.029751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d01e39c1-73fd-4567-b6a9-bfd5e992cb43" (UID: "d01e39c1-73fd-4567-b6a9-bfd5e992cb43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:37:00.031679 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.031638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d01e39c1-73fd-4567-b6a9-bfd5e992cb43" (UID: "d01e39c1-73fd-4567-b6a9-bfd5e992cb43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:37:00.031904 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.031881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d01e39c1-73fd-4567-b6a9-bfd5e992cb43" (UID: "d01e39c1-73fd-4567-b6a9-bfd5e992cb43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:37:00.031992 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.031932    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-kube-api-access-26n8r" (OuterVolumeSpecName: "kube-api-access-26n8r") pod "d01e39c1-73fd-4567-b6a9-bfd5e992cb43" (UID: "d01e39c1-73fd-4567-b6a9-bfd5e992cb43"). InnerVolumeSpecName "kube-api-access-26n8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:37:00.130438 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.130399    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-maas-api-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:00.130438 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.130431    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26n8r\" (UniqueName: \"kubernetes.io/projected/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-kube-api-access-26n8r\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:00.130438 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.130442    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-openshift-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:00.130713 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.130451    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d01e39c1-73fd-4567-b6a9-bfd5e992cb43-metrics-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:00.276074 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.275982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:00.376968 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.376938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:00.562342 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.562258    2577 generic.go:356] "Generic (PLEG): container finished" podID="d01e39c1-73fd-4567-b6a9-bfd5e992cb43" containerID="f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422" exitCode=0
Sep 16 12:37:00.562342 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.562305    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"
Sep 16 12:37:00.562517 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.562349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b" event={"ID":"d01e39c1-73fd-4567-b6a9-bfd5e992cb43","Type":"ContainerDied","Data":"f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422"}
Sep 16 12:37:00.562517 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.562398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b" event={"ID":"d01e39c1-73fd-4567-b6a9-bfd5e992cb43","Type":"ContainerDied","Data":"f55160e278ab9564aa03207025b7af0ee3a2af097420cd7dbf89837b7e7d1d26"}
Sep 16 12:37:00.562517 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.562420    2577 scope.go:117] "RemoveContainer" containerID="f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422"
Sep 16 12:37:00.571679 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.571661    2577 scope.go:117] "RemoveContainer" containerID="f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422"
Sep 16 12:37:00.571975 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:37:00.571953    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422\": container with ID starting with f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422 not found: ID does not exist" containerID="f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422"
Sep 16 12:37:00.572047 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.571984    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422"} err="failed to get container status \"f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422\": rpc error: code = NotFound desc = could not find container \"f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422\": container with ID starting with f03314e4cc62a54fbb01f7c849989bc9188e3f1b8e69c9db6f01d4ca099e3422 not found: ID does not exist"
Sep 16 12:37:00.583667 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.583638    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"]
Sep 16 12:37:00.587416 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:00.587391    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c03953-75bdf89bf5-srx4b"]
Sep 16 12:37:01.111227 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:01.111194    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d01e39c1-73fd-4567-b6a9-bfd5e992cb43" path="/var/lib/kubelet/pods/d01e39c1-73fd-4567-b6a9-bfd5e992cb43/volumes"
Sep 16 12:37:02.296526 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:02.296493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:37:06.615669 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:06.615636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:06.715991 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:06.715944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:06.892189 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:06.892159    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"]
Sep 16 12:37:06.892457 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:06.892427    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" containerID="cri-o://bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd" gracePeriod=30
Sep 16 12:37:06.914579 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:06.914532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:37:10.888882 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:10.888836    2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:37:10.889317 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:10.888924    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:37:12.157726 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.157666    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:37:12.236062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-envoy\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.236062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istiod-ca-cert\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.236310 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-credential-socket\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-certs\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btlh4\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-kube-api-access-btlh4\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236734    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-token\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-socket\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236851    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-data\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236887    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-podinfo\") pod \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\" (UID: \"97fbd44d-86b0-46d4-82e9-2ee553202e5b\") "
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.236979    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.237006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:12.237292 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.237146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-data" (OuterVolumeSpecName: "istio-data") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:12.239933 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.239912    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istiod-ca-cert\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.240072 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.240059    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-credential-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.240172 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.240160    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.240255 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.240245    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-workload-socket\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.240348 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.240336    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-data\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.245871 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.245843    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:37:12.245871 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.245851    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-kube-api-access-btlh4" (OuterVolumeSpecName: "kube-api-access-btlh4") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "kube-api-access-btlh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:37:12.246013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.245864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:37:12.246013 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.245851    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-token" (OuterVolumeSpecName: "istio-token") pod "97fbd44d-86b0-46d4-82e9-2ee553202e5b" (UID: "97fbd44d-86b0-46d4-82e9-2ee553202e5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:37:12.341482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.341440    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-envoy\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.341482 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.341487    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btlh4\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-kube-api-access-btlh4\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.341703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.341501    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-token\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.341703 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.341515    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97fbd44d-86b0-46d4-82e9-2ee553202e5b-istio-podinfo\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:37:12.618083 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.617995    2577 generic.go:356] "Generic (PLEG): container finished" podID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerID="bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd" exitCode=0
Sep 16 12:37:12.618083 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.618059    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"
Sep 16 12:37:12.618254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.618079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" event={"ID":"97fbd44d-86b0-46d4-82e9-2ee553202e5b","Type":"ContainerDied","Data":"bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd"}
Sep 16 12:37:12.618254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.618159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc" event={"ID":"97fbd44d-86b0-46d4-82e9-2ee553202e5b","Type":"ContainerDied","Data":"3dbe8fae5487a905d47c7c312ec0b69b32b5f490b44c12abf30b68d54c39b987"}
Sep 16 12:37:12.618254 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.618185    2577 scope.go:117] "RemoveContainer" containerID="bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd"
Sep 16 12:37:12.627828 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.627805    2577 scope.go:117] "RemoveContainer" containerID="cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b"
Sep 16 12:37:12.635954 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.635937    2577 scope.go:117] "RemoveContainer" containerID="bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd"
Sep 16 12:37:12.636200 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:37:12.636181    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd\": container with ID starting with bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd not found: ID does not exist" containerID="bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd"
Sep 16 12:37:12.636274 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.636214    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd"} err="failed to get container status \"bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd\": rpc error: code = NotFound desc = could not find container \"bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd\": container with ID starting with bde6e4bcfc8172921b58d990439c4022c8aa6ed424dac0d36f66ac95b7e2d2dd not found: ID does not exist"
Sep 16 12:37:12.636274 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.636238    2577 scope.go:117] "RemoveContainer" containerID="cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b"
Sep 16 12:37:12.636542 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:37:12.636522    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b\": container with ID starting with cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b not found: ID does not exist" containerID="cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b"
Sep 16 12:37:12.636596 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.636547    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b"} err="failed to get container status \"cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b\": rpc error: code = NotFound desc = could not find container \"cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b\": container with ID starting with cca2c295a8260e9346d233ecb56902be989cfd12d7444f28d9ba68a35b55b96b not found: ID does not exist"
Sep 16 12:37:12.639619 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.639598    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"]
Sep 16 12:37:12.644050 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:12.644028    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c03953-openshift-default-79cd55b754-zbjxc"]
Sep 16 12:37:13.111504 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:13.111463    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" path="/var/lib/kubelet/pods/97fbd44d-86b0-46d4-82e9-2ee553202e5b/volumes"
Sep 16 12:37:13.399761 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:37:13.399728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:38:35.975241 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:38:35.975205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:38:36.081868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:38:36.081832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:38:37.599318 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:38:37.599285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:00.080218 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:00.080182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:39:01.175939 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:01.175891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:07.995941 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:07.995902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:39:10.003192 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:10.003159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:10.104089 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:10.104057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:16.502415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:16.502339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:39:21.403769 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:21.403733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:21.504504 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:21.504471    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:39:32.182715 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:32.182669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:39:38.089696 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:38.089656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:39:39.667528 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:39.667476    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"]
Sep 16 12:39:39.668104 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:39.667702    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc" podUID="aae2e741-85e7-4c95-9a78-0da2d617e8a9" containerName="manager" containerID="cri-o://56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a" gracePeriod=10
Sep 16 12:39:40.716251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:40.716220    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:39:40.820485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:40.820407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4tn8\" (UniqueName: \"kubernetes.io/projected/aae2e741-85e7-4c95-9a78-0da2d617e8a9-kube-api-access-p4tn8\") pod \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") "
Sep 16 12:39:40.820485 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:40.820472    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae2e741-85e7-4c95-9a78-0da2d617e8a9-extensions-socket-volume\") pod \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\" (UID: \"aae2e741-85e7-4c95-9a78-0da2d617e8a9\") "
Sep 16 12:39:40.820898 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:40.820871    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae2e741-85e7-4c95-9a78-0da2d617e8a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aae2e741-85e7-4c95-9a78-0da2d617e8a9" (UID: "aae2e741-85e7-4c95-9a78-0da2d617e8a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:39:40.822427 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:40.822405    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae2e741-85e7-4c95-9a78-0da2d617e8a9-kube-api-access-p4tn8" (OuterVolumeSpecName: "kube-api-access-p4tn8") pod "aae2e741-85e7-4c95-9a78-0da2d617e8a9" (UID: "aae2e741-85e7-4c95-9a78-0da2d617e8a9"). InnerVolumeSpecName "kube-api-access-p4tn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:40.921501 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:40.921466    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae2e741-85e7-4c95-9a78-0da2d617e8a9-extensions-socket-volume\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:39:40.921501 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:40.921496    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tn8\" (UniqueName: \"kubernetes.io/projected/aae2e741-85e7-4c95-9a78-0da2d617e8a9-kube-api-access-p4tn8\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:39:41.268864 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.268828    2577 generic.go:356] "Generic (PLEG): container finished" podID="aae2e741-85e7-4c95-9a78-0da2d617e8a9" containerID="56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a" exitCode=0
Sep 16 12:39:41.269052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.268886    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"
Sep 16 12:39:41.269052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.268910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc" event={"ID":"aae2e741-85e7-4c95-9a78-0da2d617e8a9","Type":"ContainerDied","Data":"56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a"}
Sep 16 12:39:41.269052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.268965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc" event={"ID":"aae2e741-85e7-4c95-9a78-0da2d617e8a9","Type":"ContainerDied","Data":"0ecd9e1314118e5be2c40ba2e8c43f5d36987f5ae5955dfdc792ea189b13fb0c"}
Sep 16 12:39:41.269052 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.268986    2577 scope.go:117] "RemoveContainer" containerID="56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a"
Sep 16 12:39:41.277549 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.277529    2577 scope.go:117] "RemoveContainer" containerID="56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a"
Sep 16 12:39:41.277819 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:39:41.277800    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a\": container with ID starting with 56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a not found: ID does not exist" containerID="56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a"
Sep 16 12:39:41.277886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.277830    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a"} err="failed to get container status \"56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a\": rpc error: code = NotFound desc = could not find container \"56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a\": container with ID starting with 56075937cce6b473ddc927381ce233a53af1e37bce41e59b54c2287245ea6a9a not found: ID does not exist"
Sep 16 12:39:41.289954 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.289925    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"]
Sep 16 12:39:41.296558 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:41.296538    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wlkdc"]
Sep 16 12:39:43.111281 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:39:43.111247    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aae2e741-85e7-4c95-9a78-0da2d617e8a9" path="/var/lib/kubelet/pods/aae2e741-85e7-4c95-9a78-0da2d617e8a9/volumes"
Sep 16 12:40:40.377896 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.377861    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"]
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378232    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" containerName="payload-pre-processing"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378245    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" containerName="payload-pre-processing"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378256    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957908ce-a038-41f4-8da0-e55af23d54e5" containerName="maas-api"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378265    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="957908ce-a038-41f4-8da0-e55af23d54e5" containerName="maas-api"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378284    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae2e741-85e7-4c95-9a78-0da2d617e8a9" containerName="manager"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378291    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae2e741-85e7-4c95-9a78-0da2d617e8a9" containerName="manager"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378297    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e960d15-7723-4b57-8312-95ac76ca17ef" containerName="payload-processing"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378304    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e960d15-7723-4b57-8312-95ac76ca17ef" containerName="payload-processing"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378310    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378315    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378320    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d01e39c1-73fd-4567-b6a9-bfd5e992cb43" containerName="maas-api"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378326    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d01e39c1-73fd-4567-b6a9-bfd5e992cb43" containerName="maas-api"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378339    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d71e92-935b-4eb6-abe3-0781b77a8c62" containerName="payload-processing"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378344    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d71e92-935b-4eb6-abe3-0781b77a8c62" containerName="payload-processing"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378350    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b83d408-a93d-48b4-bd86-278049f0c475" containerName="payload-processing"
Sep 16 12:40:40.378362 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378359    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b83d408-a93d-48b4-bd86-278049f0c475" containerName="payload-processing"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378369    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378376    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378431    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="957908ce-a038-41f4-8da0-e55af23d54e5" containerName="maas-api"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378443    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b83d408-a93d-48b4-bd86-278049f0c475" containerName="payload-processing"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378451    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e960d15-7723-4b57-8312-95ac76ca17ef" containerName="payload-processing"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378466    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97fbd44d-86b0-46d4-82e9-2ee553202e5b" containerName="istio-proxy"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378471    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d01e39c1-73fd-4567-b6a9-bfd5e992cb43" containerName="maas-api"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378475    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aae2e741-85e7-4c95-9a78-0da2d617e8a9" containerName="manager"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378480    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bdc7512-1ec8-4d2a-b6a7-ba1a99cc8754" containerName="payload-pre-processing"
Sep 16 12:40:40.378886 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.378485    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d71e92-935b-4eb6-abe3-0781b77a8c62" containerName="payload-processing"
Sep 16 12:40:40.381294 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.381277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.387480 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.387461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ktrdp\""
Sep 16 12:40:40.403472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.403446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"]
Sep 16 12:40:40.505475 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.505440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ea26f86-27da-41c8-968f-cfb586beea63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8fcq8\" (UID: \"9ea26f86-27da-41c8-968f-cfb586beea63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.505676 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.505511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzrj\" (UniqueName: \"kubernetes.io/projected/9ea26f86-27da-41c8-968f-cfb586beea63-kube-api-access-kbzrj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8fcq8\" (UID: \"9ea26f86-27da-41c8-968f-cfb586beea63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.606704 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.606664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzrj\" (UniqueName: \"kubernetes.io/projected/9ea26f86-27da-41c8-968f-cfb586beea63-kube-api-access-kbzrj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8fcq8\" (UID: \"9ea26f86-27da-41c8-968f-cfb586beea63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.606875 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.606737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ea26f86-27da-41c8-968f-cfb586beea63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8fcq8\" (UID: \"9ea26f86-27da-41c8-968f-cfb586beea63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.607151 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.607131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ea26f86-27da-41c8-968f-cfb586beea63-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8fcq8\" (UID: \"9ea26f86-27da-41c8-968f-cfb586beea63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.616545 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.616513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzrj\" (UniqueName: \"kubernetes.io/projected/9ea26f86-27da-41c8-968f-cfb586beea63-kube-api-access-kbzrj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-8fcq8\" (UID: \"9ea26f86-27da-41c8-968f-cfb586beea63\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.690699 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.690668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:40.967725 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:40.967700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"]
Sep 16 12:40:40.969770 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:40:40.969741    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea26f86_27da_41c8_968f_cfb586beea63.slice/crio-ee138e1d6a084abb92912405fb3141263a13cb0fa1d242e4a37470efec1920d8 WatchSource:0}: Error finding container ee138e1d6a084abb92912405fb3141263a13cb0fa1d242e4a37470efec1920d8: Status 404 returned error can't find the container with id ee138e1d6a084abb92912405fb3141263a13cb0fa1d242e4a37470efec1920d8
Sep 16 12:40:41.526938 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:41.526899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8" event={"ID":"9ea26f86-27da-41c8-968f-cfb586beea63","Type":"ContainerStarted","Data":"59ab7ef4fc420b9b7f270be45184f1acddbc880a19cc0ae011d568b35c104f44"}
Sep 16 12:40:41.527431 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:41.526964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8" event={"ID":"9ea26f86-27da-41c8-968f-cfb586beea63","Type":"ContainerStarted","Data":"ee138e1d6a084abb92912405fb3141263a13cb0fa1d242e4a37470efec1920d8"}
Sep 16 12:40:41.527431 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:41.527010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:40:41.555141 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:41.555047    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8" podStartSLOduration=1.5550280170000002 podStartE2EDuration="1.555028017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:41.553651241 +0000 UTC m=+2328.965668138" watchObservedRunningTime="2026-09-16 12:40:41.555028017 +0000 UTC m=+2328.967044914"
Sep 16 12:40:47.215016 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:47.214929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:47.501356 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:47.501272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:40:52.533617 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:40:52.533581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-8fcq8"
Sep 16 12:41:03.804670 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:03.804630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:41:07.097895 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:07.097857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:07.843019 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:07.842978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:11.300756 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:11.300709    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:41:19.310037 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:19.309994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:41:26.387868 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:26.387829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:28.903000 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:28.902968    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:41:34.398652 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:34.398619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:37.332551 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:37.332515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:41:50.818820 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:50.818735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:52.733876 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:52.733840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:41:53.018999 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:41:53.018919    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:01.722944 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:01.722902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:09.340893 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:09.340850    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:16.015047 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.014960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:16.356279 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.356184    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"]
Sep 16 12:42:16.359553 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.359529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.363819 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.363796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:42:16.363819 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.363810    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:42:16.364235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.364217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d7fbn\""
Sep 16 12:42:16.364291 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.364226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:42:16.364352 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.364332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 12:42:16.382819 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.382771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"]
Sep 16 12:42:16.429792 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.429756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8znjt\" (UniqueName: \"kubernetes.io/projected/417bd723-a42d-42e5-87e3-3f4058a4d6bd-kube-api-access-8znjt\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.429960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.429863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-metrics-certs\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.429960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.429894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-maas-api-tls\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.429960 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.429921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd723-a42d-42e5-87e3-3f4058a4d6bd-openshift-service-ca\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.531302 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.531258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-metrics-certs\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.531497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.531309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-maas-api-tls\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.531497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.531340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd723-a42d-42e5-87e3-3f4058a4d6bd-openshift-service-ca\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.531497 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.531401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8znjt\" (UniqueName: \"kubernetes.io/projected/417bd723-a42d-42e5-87e3-3f4058a4d6bd-kube-api-access-8znjt\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.532224 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.532194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd723-a42d-42e5-87e3-3f4058a4d6bd-openshift-service-ca\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.533903 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.533882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-maas-api-tls\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.534062 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.534041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-metrics-certs\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.546928 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.546906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8znjt\" (UniqueName: \"kubernetes.io/projected/417bd723-a42d-42e5-87e3-3f4058a4d6bd-kube-api-access-8znjt\") pod \"maas-api-6cd6bf8c7c-fhnl6\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") " pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.669914 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.669873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:16.953855 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.953761    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"]
Sep 16 12:42:16.956758 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:42:16.956730    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417bd723_a42d_42e5_87e3_3f4058a4d6bd.slice/crio-10782aff30d83b7d804eec1f096fc98d8038e7a7a73efa4bfbea9e2d70779111 WatchSource:0}: Error finding container 10782aff30d83b7d804eec1f096fc98d8038e7a7a73efa4bfbea9e2d70779111: Status 404 returned error can't find the container with id 10782aff30d83b7d804eec1f096fc98d8038e7a7a73efa4bfbea9e2d70779111
Sep 16 12:42:16.958299 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:16.958279    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:42:17.963854 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:17.963815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6" event={"ID":"417bd723-a42d-42e5-87e3-3f4058a4d6bd","Type":"ContainerStarted","Data":"4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397"}
Sep 16 12:42:17.964326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:17.963866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6" event={"ID":"417bd723-a42d-42e5-87e3-3f4058a4d6bd","Type":"ContainerStarted","Data":"10782aff30d83b7d804eec1f096fc98d8038e7a7a73efa4bfbea9e2d70779111"}
Sep 16 12:42:17.964326 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:17.963939    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:17.996890 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:17.996834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6" podStartSLOduration=1.6143628300000001 podStartE2EDuration="1.996817241s" totalImagesPullingTime="382.454411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:17.996808972 +0000 UTC m=+2425.408825871" watchObservedRunningTime="2026-09-16 12:42:17.996817241 +0000 UTC m=+2425.408834136"
Sep 16 12:42:23.978729 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:23.978699    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:24.117488 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.117458    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:24.246815 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.246735    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"]
Sep 16 12:42:24.250116 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.250099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.259480 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.259451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"]
Sep 16 12:42:24.298026 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.297995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798ba607-8e4a-4385-af58-54938b4a5a75-metrics-certs\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.298157 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.298063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798ba607-8e4a-4385-af58-54938b4a5a75-maas-api-tls\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.298157 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.298111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wvj\" (UniqueName: \"kubernetes.io/projected/798ba607-8e4a-4385-af58-54938b4a5a75-kube-api-access-j7wvj\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.298157 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.298134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798ba607-8e4a-4385-af58-54938b4a5a75-openshift-service-ca\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.399499 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.399470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798ba607-8e4a-4385-af58-54938b4a5a75-maas-api-tls\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.399707 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.399525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wvj\" (UniqueName: \"kubernetes.io/projected/798ba607-8e4a-4385-af58-54938b4a5a75-kube-api-access-j7wvj\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.399707 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.399637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798ba607-8e4a-4385-af58-54938b4a5a75-openshift-service-ca\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.399831 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.399719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798ba607-8e4a-4385-af58-54938b4a5a75-metrics-certs\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.400283 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.400259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/798ba607-8e4a-4385-af58-54938b4a5a75-openshift-service-ca\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.402167 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.402145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/798ba607-8e4a-4385-af58-54938b4a5a75-maas-api-tls\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.402276 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.402177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798ba607-8e4a-4385-af58-54938b4a5a75-metrics-certs\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.415251 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.415230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wvj\" (UniqueName: \"kubernetes.io/projected/798ba607-8e4a-4385-af58-54938b4a5a75-kube-api-access-j7wvj\") pod \"maas-api-88dfc4b4f-nhzw9\" (UID: \"798ba607-8e4a-4385-af58-54938b4a5a75\") " pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.562237 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.562148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:24.850422 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:24.850396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"]
Sep 16 12:42:24.852888 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:42:24.852850    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798ba607_8e4a_4385_af58_54938b4a5a75.slice/crio-1f1e4e52226ef0df2837b02694b1123af71c9bb775195952decf6f0acb1d17ff WatchSource:0}: Error finding container 1f1e4e52226ef0df2837b02694b1123af71c9bb775195952decf6f0acb1d17ff: Status 404 returned error can't find the container with id 1f1e4e52226ef0df2837b02694b1123af71c9bb775195952decf6f0acb1d17ff
Sep 16 12:42:25.001957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:25.001919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9" event={"ID":"798ba607-8e4a-4385-af58-54938b4a5a75","Type":"ContainerStarted","Data":"1f1e4e52226ef0df2837b02694b1123af71c9bb775195952decf6f0acb1d17ff"}
Sep 16 12:42:26.008829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:26.008786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9" event={"ID":"798ba607-8e4a-4385-af58-54938b4a5a75","Type":"ContainerStarted","Data":"0ea308b61714ac781ffb87e2b53cd2bd1fc03aae88672b0025225ff4b48f1c8d"}
Sep 16 12:42:26.009280 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:26.008948    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:26.053360 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:26.053302    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9" podStartSLOduration=1.48696071 podStartE2EDuration="2.053285074s" totalImagesPullingTime="566.324364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:26.050370312 +0000 UTC m=+2433.462387209" watchObservedRunningTime="2026-09-16 12:42:26.053285074 +0000 UTC m=+2433.465301989"
Sep 16 12:42:27.127923 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:27.127883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:42:31.056449 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:31.056414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:32.019432 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.019391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-88dfc4b4f-nhzw9"
Sep 16 12:42:32.147465 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.147433    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"]
Sep 16 12:42:32.147842 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.147621    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6" podUID="417bd723-a42d-42e5-87e3-3f4058a4d6bd" containerName="maas-api" containerID="cri-o://4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397" gracePeriod=30
Sep 16 12:42:32.388435 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.388412    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:32.467235 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.467197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8znjt\" (UniqueName: \"kubernetes.io/projected/417bd723-a42d-42e5-87e3-3f4058a4d6bd-kube-api-access-8znjt\") pod \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") "
Sep 16 12:42:32.467522 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.467254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd723-a42d-42e5-87e3-3f4058a4d6bd-openshift-service-ca\") pod \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") "
Sep 16 12:42:32.467522 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.467290    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-maas-api-tls\") pod \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") "
Sep 16 12:42:32.467522 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.467327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-metrics-certs\") pod \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\" (UID: \"417bd723-a42d-42e5-87e3-3f4058a4d6bd\") "
Sep 16 12:42:32.467749 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.467703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417bd723-a42d-42e5-87e3-3f4058a4d6bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "417bd723-a42d-42e5-87e3-3f4058a4d6bd" (UID: "417bd723-a42d-42e5-87e3-3f4058a4d6bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:32.469378 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.469352    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417bd723-a42d-42e5-87e3-3f4058a4d6bd-kube-api-access-8znjt" (OuterVolumeSpecName: "kube-api-access-8znjt") pod "417bd723-a42d-42e5-87e3-3f4058a4d6bd" (UID: "417bd723-a42d-42e5-87e3-3f4058a4d6bd"). InnerVolumeSpecName "kube-api-access-8znjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:32.469496 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.469475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "417bd723-a42d-42e5-87e3-3f4058a4d6bd" (UID: "417bd723-a42d-42e5-87e3-3f4058a4d6bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:32.469561 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.469491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "417bd723-a42d-42e5-87e3-3f4058a4d6bd" (UID: "417bd723-a42d-42e5-87e3-3f4058a4d6bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:32.568829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.568749    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8znjt\" (UniqueName: \"kubernetes.io/projected/417bd723-a42d-42e5-87e3-3f4058a4d6bd-kube-api-access-8znjt\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:42:32.568829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.568779    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/417bd723-a42d-42e5-87e3-3f4058a4d6bd-openshift-service-ca\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:42:32.568829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.568791    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-maas-api-tls\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:42:32.568829 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:32.568800    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/417bd723-a42d-42e5-87e3-3f4058a4d6bd-metrics-certs\") on node \"ip-10-0-133-38.ec2.internal\" DevicePath \"\""
Sep 16 12:42:33.041786 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.041754    2577 generic.go:356] "Generic (PLEG): container finished" podID="417bd723-a42d-42e5-87e3-3f4058a4d6bd" containerID="4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397" exitCode=0
Sep 16 12:42:33.041965 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.041799    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"
Sep 16 12:42:33.041965 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.041841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6" event={"ID":"417bd723-a42d-42e5-87e3-3f4058a4d6bd","Type":"ContainerDied","Data":"4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397"}
Sep 16 12:42:33.041965 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.041888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6" event={"ID":"417bd723-a42d-42e5-87e3-3f4058a4d6bd","Type":"ContainerDied","Data":"10782aff30d83b7d804eec1f096fc98d8038e7a7a73efa4bfbea9e2d70779111"}
Sep 16 12:42:33.041965 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.041910    2577 scope.go:117] "RemoveContainer" containerID="4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397"
Sep 16 12:42:33.051722 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.051702    2577 scope.go:117] "RemoveContainer" containerID="4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397"
Sep 16 12:42:33.051976 ip-10-0-133-38 kubenswrapper[2577]: E0916 12:42:33.051958    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397\": container with ID starting with 4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397 not found: ID does not exist" containerID="4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397"
Sep 16 12:42:33.052021 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.051983    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397"} err="failed to get container status \"4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397\": rpc error: code = NotFound desc = could not find container \"4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397\": container with ID starting with 4260d5bf203778b0ae36d353f24029baf32c7617279e17d838f0859031ed0397 not found: ID does not exist"
Sep 16 12:42:33.082284 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.082253    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"]
Sep 16 12:42:33.099067 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.099040    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cd6bf8c7c-fhnl6"]
Sep 16 12:42:33.111507 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:33.111482    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="417bd723-a42d-42e5-87e3-3f4058a4d6bd" path="/var/lib/kubelet/pods/417bd723-a42d-42e5-87e3-3f4058a4d6bd/volumes"
Sep 16 12:42:37.308169 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:37.308127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:43.505513 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:43.505474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dqgg6"]
Sep 16 12:42:53.774472 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:53.774441    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vz5nf_50418aa1-661f-4523-b5d3-031f0448be72/server/0.log"
Sep 16 12:42:53.925804 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:53.925765    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8ps7n_44725276-0e2d-4c99-b373-748fcf0ab7bc/manager/0.log"
Sep 16 12:42:54.059351 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:54.059321    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-mlx69_7189d010-0c01-4b6d-be71-a0c45572fcf2/copy-manifests/0.log"
Sep 16 12:42:54.085069 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:54.085040    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-mlx69_7189d010-0c01-4b6d-be71-a0c45572fcf2/manager/0.log"
Sep 16 12:42:54.901376 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:54.901338    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-88dfc4b4f-nhzw9_798ba607-8e4a-4385-af58-54938b4a5a75/maas-api/0.log"
Sep 16 12:42:55.015937 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:55.015907    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826015-hmmpf_3fbff420-1683-480f-88bc-b5015ba682ec/cleanup/0.log"
Sep 16 12:42:56.841266 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:56.841233    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-8fcq8_9ea26f86-27da-41c8-968f-cfb586beea63/manager/0.log"
Sep 16 12:42:57.067157 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:57.067123    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dqgg6_ea5973c9-115f-4a33-bedb-933f2b48e239/limitador/0.log"
Sep 16 12:42:57.220926 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:57.220890    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-wppcw_f6370efc-3bc9-4820-9f7a-a636521d1c01/manager/0.log"
Sep 16 12:42:57.752475 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:57.752443    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dlp49_f51eb83f-4f25-4a0a-8603-95d2f58e5632/discovery/0.log"
Sep 16 12:42:58.024527 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:58.024406    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hvfvl_7970c6ed-7aa3-422a-88b7-f2491a98e453/kube-auth-proxy/0.log"
Sep 16 12:42:59.383300 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:59.383272    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6q84w_a514860c-5bb5-4d0f-94c2-b1b45f5a3efb/main/0.log"
Sep 16 12:42:59.631991 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:42:59.631950    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-plrcp_65b471c6-472e-480f-9617-a6ebaa6cf1dc/main/0.log"
Sep 16 12:43:06.231344 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:06.231307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:43:06.426925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:06.426891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log" limit=1024
Sep 16 12:43:09.931250 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:09.931216    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lc92b_bdf83e47-967c-46a9-8e56-ac2a7b147088/global-pull-secret-syncer/0.log"
Sep 16 12:43:10.010181 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:10.010148    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h24wk_0aba3353-392c-4424-a645-ec6ea7d72619/konnectivity-agent/0.log"
Sep 16 12:43:10.124668 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:10.124632    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-38.ec2.internal_a3571c7ce18de9915aa1b3a13a6c6bc9/haproxy/0.log"
Sep 16 12:43:15.531047 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:15.531009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-8fcq8_9ea26f86-27da-41c8-968f-cfb586beea63/manager/0.log"
Sep 16 12:43:15.615467 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:15.615438    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dqgg6_ea5973c9-115f-4a33-bedb-933f2b48e239/limitador/0.log"
Sep 16 12:43:15.729646 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:15.729607    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-wppcw_f6370efc-3bc9-4820-9f7a-a636521d1c01/manager/0.log"
Sep 16 12:43:18.361373 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.361344    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-64qhf_b37dc13c-89f9-4f1d-8f1b-80924302c0e1/node-exporter/0.log"
Sep 16 12:43:18.387855 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.387828    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-64qhf_b37dc13c-89f9-4f1d-8f1b-80924302c0e1/kube-rbac-proxy/0.log"
Sep 16 12:43:18.418051 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.418021    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-64qhf_b37dc13c-89f9-4f1d-8f1b-80924302c0e1/init-textfile/0.log"
Sep 16 12:43:18.916584 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.916532    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"]
Sep 16 12:43:18.916905 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.916890    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417bd723-a42d-42e5-87e3-3f4058a4d6bd" containerName="maas-api"
Sep 16 12:43:18.916964 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.916906    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="417bd723-a42d-42e5-87e3-3f4058a4d6bd" containerName="maas-api"
Sep 16 12:43:18.917000 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.916969    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="417bd723-a42d-42e5-87e3-3f4058a4d6bd" containerName="maas-api"
Sep 16 12:43:18.919984 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.919968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:18.924412 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.924394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8nlh7\"/\"kube-root-ca.crt\""
Sep 16 12:43:18.924540 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.924390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8nlh7\"/\"default-dockercfg-x94cz\""
Sep 16 12:43:18.925264 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.925235    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8nlh7\"/\"openshift-service-ca.crt\""
Sep 16 12:43:18.952412 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.952380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"]
Sep 16 12:43:18.954010 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.953984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-sys\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:18.954144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.954019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvr5\" (UniqueName: \"kubernetes.io/projected/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-kube-api-access-fhvr5\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:18.954144 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.954042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-lib-modules\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:18.954263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.954165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-podres\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:18.954263 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:18.954201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-proc\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.055717 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-proc\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.055889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-sys\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.055889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvr5\" (UniqueName: \"kubernetes.io/projected/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-kube-api-access-fhvr5\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.055889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-lib-modules\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.055889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-proc\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.055889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-podres\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.055889 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-sys\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.056130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055928    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-lib-modules\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.056130 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.055946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-podres\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.070771 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.070752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvr5\" (UniqueName: \"kubernetes.io/projected/c10c2abc-e3a5-4c28-a367-95c2ff39d5e1-kube-api-access-fhvr5\") pod \"perf-node-gather-daemonset-rgfrt\" (UID: \"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1\") " pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.229925 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.229826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:19.516927 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:19.516840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"]
Sep 16 12:43:19.521349 ip-10-0-133-38 kubenswrapper[2577]: W0916 12:43:19.521322    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc10c2abc_e3a5_4c28_a367_95c2ff39d5e1.slice/crio-6c4f9157a9308043dc3eb5ca788743e20dfe41a7771b6cc8a31ced1597810a6f WatchSource:0}: Error finding container 6c4f9157a9308043dc3eb5ca788743e20dfe41a7771b6cc8a31ced1597810a6f: Status 404 returned error can't find the container with id 6c4f9157a9308043dc3eb5ca788743e20dfe41a7771b6cc8a31ced1597810a6f
Sep 16 12:43:20.267491 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:20.267457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt" event={"ID":"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1","Type":"ContainerStarted","Data":"46a7729d92f942c17099a8cba5d66944a5cb8bb37c920420b79d27a5cdc1bbd1"}
Sep 16 12:43:20.267910 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:20.267508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt" event={"ID":"c10c2abc-e3a5-4c28-a367-95c2ff39d5e1","Type":"ContainerStarted","Data":"6c4f9157a9308043dc3eb5ca788743e20dfe41a7771b6cc8a31ced1597810a6f"}
Sep 16 12:43:20.267910 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:20.267602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:20.840643 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:20.840613    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-c4qlf_a467175a-cb94-403f-a278-ce1af1396e85/networking-console-plugin/0.log"
Sep 16 12:43:23.788415 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:23.788386    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vsqll_832aad20-0733-43cf-9a64-e417f6d13bdb/dns/0.log"
Sep 16 12:43:23.811994 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:23.811970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vsqll_832aad20-0733-43cf-9a64-e417f6d13bdb/kube-rbac-proxy/0.log"
Sep 16 12:43:23.837715 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:23.837695    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2s4xk_4942bb0d-de9b-4af1-8ece-2d67d2a91f51/dns-node-resolver/0.log"
Sep 16 12:43:24.544802 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:24.544771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6958b676fb-fvx7b_d8cc093d-c95e-463e-bba0-c9f1b57a2ac5/registry/0.log"
Sep 16 12:43:24.595716 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:24.595684    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cvcd6_db0b38f7-df0b-4486-9151-97c4fe311900/node-ca/0.log"
Sep 16 12:43:26.051598 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:26.051553    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dlp49_f51eb83f-4f25-4a0a-8603-95d2f58e5632/discovery/0.log"
Sep 16 12:43:26.098343 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:26.098320    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hvfvl_7970c6ed-7aa3-422a-88b7-f2491a98e453/kube-auth-proxy/0.log"
Sep 16 12:43:26.282301 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:26.282275    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt"
Sep 16 12:43:26.302590 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:26.302485    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8nlh7/perf-node-gather-daemonset-rgfrt" podStartSLOduration=8.302470591 podStartE2EDuration="8.302470591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:43:20.291963219 +0000 UTC m=+2487.703980114" watchObservedRunningTime="2026-09-16 12:43:26.302470591 +0000 UTC m=+2493.714487487"
Sep 16 12:43:27.135756 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:27.135727    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jqslb_f193c68b-22e0-47b7-bad7-a883c871948a/serve-healthcheck-canary/0.log"
Sep 16 12:43:27.778337 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:27.778310    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22d4w_df95d8a0-b180-405e-ac4b-7b1d44a792db/kube-rbac-proxy/0.log"
Sep 16 12:43:27.803937 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:27.803911    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22d4w_df95d8a0-b180-405e-ac4b-7b1d44a792db/exporter/0.log"
Sep 16 12:43:27.828637 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:27.828615    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22d4w_df95d8a0-b180-405e-ac4b-7b1d44a792db/extractor/0.log"
Sep 16 12:43:30.982964 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:30.982935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vz5nf_50418aa1-661f-4523-b5d3-031f0448be72/server/0.log"
Sep 16 12:43:31.103349 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:31.103315    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8ps7n_44725276-0e2d-4c99-b373-748fcf0ab7bc/manager/0.log"
Sep 16 12:43:31.202321 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:31.202285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-mlx69_7189d010-0c01-4b6d-be71-a0c45572fcf2/manager/0.log"
Sep 16 12:43:31.234214 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:31.234128    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b54c497cf-mlx69_7189d010-0c01-4b6d-be71-a0c45572fcf2/copy-manifests/0.log"
Sep 16 12:43:31.356163 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:31.356131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dlp49_f51eb83f-4f25-4a0a-8603-95d2f58e5632/discovery/0.log"
Sep 16 12:43:31.381061 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:31.381034    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hvfvl_7970c6ed-7aa3-422a-88b7-f2491a98e453/kube-auth-proxy/0.log"
Sep 16 12:43:32.793298 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:32.793266    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cbksd_66bc3b23-d325-4ff6-b9fe-46fba1560000/openshift-lws-operator/0.log"
Sep 16 12:43:33.532073 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:33.532037    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dlp49_f51eb83f-4f25-4a0a-8603-95d2f58e5632/discovery/0.log"
Sep 16 12:43:33.575957 ip-10-0-133-38 kubenswrapper[2577]: I0916 12:43:33.575925    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d74dbbc-hvfvl_7970c6ed-7aa3-422a-88b7-f2491a98e453/kube-auth-proxy/0.log"