Jul 08 18:56:52.774876 ip-10-0-128-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 08 18:56:52.774889 ip-10-0-128-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 08 18:56:52.774899 ip-10-0-128-89 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 08 18:56:52.775221 ip-10-0-128-89 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 08 18:57:04.018522 ip-10-0-128-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 08 18:57:04.018536 ip-10-0-128-89 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dab2f076beea4bcc8d6ede6a62ef746f --
Jul 08 18:59:01.190987 ip-10-0-128-89 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.956074    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964757    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964774    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964779    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964783    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964787    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964794    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964797    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 18:59:01.968130 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964800    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964808    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964812    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964815    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964818    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964821    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964824    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964827    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964830    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964835    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964843    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964846    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964849    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964852    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964854    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964857    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964860    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964863    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964866    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964869    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 18:59:01.969532 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964873    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964877    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964880    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964885    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964888    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964891    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964895    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964899    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964902    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964905    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964908    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964911    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964914    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964917    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964923    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964926    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964929    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964932    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964935    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964938    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 18:59:01.971748 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964941    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964944    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964947    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964950    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964953    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964956    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964958    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964964    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964968    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964971    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964974    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964977    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964981    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964984    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964987    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964990    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964993    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964996    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.964999    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965002    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 18:59:01.972728 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965007    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965010    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965015    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965020    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965023    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965027    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965030    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965034    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965038    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965041    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965044    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965050    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965053    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965056    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965059    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965062    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965065    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965067    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965070    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 18:59:01.974188 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965073    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965076    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965079    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965082    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965086    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965092    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965095    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965098    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965101    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965104    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965107    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965110    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.965113    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966373    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966381    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966384    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966388    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966391    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966394    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 18:59:01.975134 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966398    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966401    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966403    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966406    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966409    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966415    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966419    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966421    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966424    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966428    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966433    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966438    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966441    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966444    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966448    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966452    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966455    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966461    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966464    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 18:59:01.976107 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966467    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966470    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966473    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966476    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966479    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966484    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966487    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966490    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966495    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966498    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966504    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966507    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966510    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966513    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966516    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966519    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966522    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966526    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966529    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 18:59:01.977073 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966532    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966535    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966539    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966545    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966549    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966552    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966555    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966558    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966561    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966564    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966567    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966570    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966573    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966576    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966578    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966581    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966587    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966589    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966592    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966595    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 18:59:01.978027 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966598    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966601    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966604    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966607    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966609    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966612    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966616    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966619    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966625    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966630    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966633    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966636    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966639    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966642    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966645    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966647    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966652    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966655    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966657    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966660    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 18:59:01.980009 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966666    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966668    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966671    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966674    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966677    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966680    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966683    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966686    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966689    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966692    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966695    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966735    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966741    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966744    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.966747    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967481    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967488    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967491    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967513    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967517    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 18:59:01.981008 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967521    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967526    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967529    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967535    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967539    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967542    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967545    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967547    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967550    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967554    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967556    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967559    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967562    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967565    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967568    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967570    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967576    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967580    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967585    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967589    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 18:59:01.981981 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967599    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967606    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967608    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967611    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967614    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967617    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967620    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967623    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967629    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967632    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967635    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967638    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967642    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967646    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967648    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967651    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967656    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967659    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967663    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967678    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 18:59:01.982956 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967684    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967687    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967691    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967694    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967697    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967700    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967703    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967707    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967709    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967712    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967715    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967718    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967726    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967732    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967735    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967738    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967741    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967746    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967750    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 18:59:01.984414 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967753    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967758    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967763    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967770    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967778    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967787    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967793    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967796    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967800    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967803    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967805    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967808    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967811    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967814    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967817    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967819    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967822    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967827    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967830    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967833    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 18:59:01.985387 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967836    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967839    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967842    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967844    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967847    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967850    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967854    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967857    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967860    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967865    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967868    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967871    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967874    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967877    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.967880    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.967998    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968007    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968014    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968023    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968028    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968031    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 18:59:01.987108 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968039    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968044    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968048    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968051    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968055    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968059    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968063    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968066    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968069    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968075    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968078    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968082    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968086    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968089    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968094    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968097    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968101    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968110    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968114    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968121    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968127    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968138    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968142    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968146    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 18:59:01.988152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968150    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968153    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968161    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968164    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968168    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968173    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968177    2563 flags.go:64] FLAG: --enable-server="true"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968180    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968187    2563 flags.go:64] FLAG: --event-burst="100"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968190    2563 flags.go:64] FLAG: --event-qps="50"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968196    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968200    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968203    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968208    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968211    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968214    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968218    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968221    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968227    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968230    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968233    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968236    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968240    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968243    2563 flags.go:64] FLAG: --feature-gates=""
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968247    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 18:59:01.989622 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968252    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968255    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968261    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968265    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968269    2563 flags.go:64] FLAG: --help="false"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968272    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-128-89.ec2.internal"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968276    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968279    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968283    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968286    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968295    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968300    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968309    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968312    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968317    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968320    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968324    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968327    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968330    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968336    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968339    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968342    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968345    2563 flags.go:64] FLAG: --lock-file=""
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968349    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 18:59:01.990845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968352    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968355    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968362    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968368    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968371    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968374    2563 flags.go:64] FLAG: --logging-format="text"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968377    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968381    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968385    2563 flags.go:64] FLAG: --manifest-url=""
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968389    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968394    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968397    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968404    2563 flags.go:64] FLAG: --max-pods="110"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968408    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968411    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968415    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968418    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968421    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968425    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968428    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968440    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968444    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968447    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 18:59:01.992559 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968451    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968455    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968462    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968466    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968478    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968488    2563 flags.go:64] FLAG: --port="10250"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968492    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968496    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00e41417c51ba4942"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968500    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968503    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968507    2563 flags.go:64] FLAG: --register-node="true"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968510    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968515    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968520    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968524    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968527    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968531    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968535    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968539    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968542    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968545    2563 flags.go:64] FLAG: --runonce="false"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968551    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968555    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968558    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968561    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 18:59:01.993993 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968564    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968568    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968571    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968575    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968580    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968585    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968589    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968592    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968596    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968600    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968603    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968612    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968615    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968618    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968622    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968626    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968629    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968632    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968635    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968639    2563 flags.go:64] FLAG: --v="2"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968649    2563 flags.go:64] FLAG: --version="false"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968656    2563 flags.go:64] FLAG: --vmodule=""
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968663    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.968667    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968922    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 18:59:01.995730 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968930    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968933    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968936    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968939    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968942    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968945    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968948    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968951    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968958    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968962    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968964    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968969    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968972    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968975    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968978    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968980    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968984    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.968987    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969020    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 18:59:01.996876 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969025    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969032    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969035    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969039    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969043    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969047    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969051    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969054    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969057    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969061    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969065    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969069    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969076    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969080    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969083    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969086    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969089    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969092    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969095    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 18:59:01.999100 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969098    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969101    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969104    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969107    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969109    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969115    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969120    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969123    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969126    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969129    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969132    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969136    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969139    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969142    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969145    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969148    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969150    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969153    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969156    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969162    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 18:59:02.000067 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969164    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969167    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969173    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969178    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969183    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969192    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969198    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969201    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969204    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969207    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969210    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969216    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969219    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969222    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969225    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969228    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969230    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969235    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969238    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 18:59:02.001044 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969241    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969244    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969247    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969250    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969253    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969259    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969262    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969265    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969269    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969272    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969274    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969278    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969281    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969283    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969286    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969289    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969292    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969297    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969300    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969304    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 18:59:02.002019 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:01.969307    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 18:59:02.003489 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:01.970100    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.027112    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.027136    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.027162    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.027169    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027239    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027246    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027250    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027253    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027257    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027260    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027263    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027266    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027269    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027273    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027276    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027280    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027283    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027286    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027290    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 18:59:02.080926 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027292    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027295    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027298    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027301    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027306    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027311    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027314    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027318    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027321    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027324    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027327    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027330    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027333    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027337    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027340    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027342    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027345    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027348    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027351    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 18:59:02.082348 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027354    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027357    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027359    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027370    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027375    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027379    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027383    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027386    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027390    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027394    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027397    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027401    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027404    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027407    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027410    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027413    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027417    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027420    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027422    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027425    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 18:59:02.083310 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027428    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027431    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027434    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027437    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027440    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027442    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027445    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027448    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027451    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027454    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027457    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027459    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027462    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027466    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027468    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027471    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027474    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027477    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027480    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027483    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 18:59:02.084993 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027487    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027489    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027492    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027495    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027498    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027501    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027504    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027507    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027511    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027515    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027518    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027521    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027524    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027527    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027530    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027533    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027535    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027538    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027541    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 18:59:02.086431 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027544    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027546    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027549    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027552    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027555    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027558    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.027564    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027699    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027725    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027729    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027734    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027739    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027742    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027745    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 18:59:02.088096 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027748    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027753    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027756    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027759    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027763    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027766    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027770    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027773    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027775    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027780    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027783    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027786    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027790    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027793    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027795    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027798    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027802    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027805    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027808    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 18:59:02.088808 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027811    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027815    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027818    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027821    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027824    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027827    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027830    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027833    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027837    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027840    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027844    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027847    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027850    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027853    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027856    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027859    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027862    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027865    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027868    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027871    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 18:59:02.090228 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027874    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027876    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027879    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027882    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027885    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027888    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027891    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027894    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027897    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027900    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027903    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027906    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027909    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027911    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027914    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027917    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027920    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027923    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027926    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027928    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 18:59:02.091183 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027931    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027934    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027936    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027939    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027942    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027945    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027948    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027950    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027953    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027956    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027959    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027962    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027968    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027971    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027974    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027977    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027980    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027984    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027988    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 18:59:02.092151 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027991    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027994    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027997    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.027999    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028004    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028007    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028010    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028012    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028015    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028018    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028021    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028024    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028027    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:02.028029    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.028035    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 18:59:02.093116 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.028185    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.031744    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.033642    2563 server.go:1019] "Starting client certificate rotation"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.033760    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.033805    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.039126    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.040580    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.061666    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.063949    2563 log.go:25] "Validated CRI v1 image API"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.065366    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.068084    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f2979854-1149-46b6-bbb8-99e116502304:/dev/nvme0n1p4 f8033e3f-36e5-451d-8cb0-f1554e17fe2a:/dev/nvme0n1p3]
Jul 08 18:59:02.093849 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.068100    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.073348    2563 manager.go:216] Machine: {Timestamp:2026-07-08 18:59:02.071976857 +0000 UTC m=+0.715997581 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104222 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27c6c662b092cb5fcfb9517c8ed787 SystemUUID:ec27c6c6-62b0-92cb-5fcf-b9517c8ed787 BootID:dab2f076-beea-4bcc-8d6e-de6a62ef746f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d8:29:f1:63:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d8:29:f1:63:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:83:65:26:a1:8f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.073461    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.073554    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.074517    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.074724    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.074759    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.075482    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.075492    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.075500    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.078169    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.079688    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.084962    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085101    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085135    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085150    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085165    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 18:59:02.095035 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085175    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 08 18:59:02.095984 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085185    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 18:59:02.095984 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085605    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 18:59:02.095984 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.085623    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.126990    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9rdz9"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.131027    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.148585    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.148627    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.149977    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.149995    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150007    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150023    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150036    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150045    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150057    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150083    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150516    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.150527    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.151973    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.127053    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9rdz9"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.152097    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.153194    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.153208    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.155750    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.155773    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.155881    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.158312    2563 server.go:1268] "Started kubelet"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.158370    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.158594    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.158671    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.160318    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.160908    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 18:59:02.174066 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.164009    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 18:59:02.159551 ip-10-0-128-89 systemd[1]: Started Kubernetes Kubelet.
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.204739    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.204858    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.204887    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205401    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205404    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205426    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205564    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205574    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205755    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205772    2563 factory.go:55] Registering systemd factory
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.205780    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.206036    2563 factory.go:153] Registering CRI-O factory
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.206052    2563 factory.go:223] Registration of the crio container factory successfully
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.206079    2563 factory.go:103] Registering Raw factory
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.206100    2563 manager.go:1196] Started watching for new ooms in manager
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.206615    2563 manager.go:349] Starting recovery of all containers
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.226076    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.226123    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.256942    2563 manager.go:354] Recovery completed
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.262607    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.264527    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.264560    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.264575    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.265170    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.265184    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.265186    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.265206    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.265961    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.265989    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 18:59:02.267561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.266002    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 18:59:02.270505 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.267736    2563 policy_none.go:49] "None policy: Start"
Jul 08 18:59:02.270505 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.267752    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 18:59:02.270505 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.267764    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 18:59:02.270505 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.269453    2563 policy_none.go:47] "Start"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.304910    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.341344    2563 manager.go:337] "Starting Device Plugin manager"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.341377    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.341387    2563 server.go:87] "Starting device plugin registration server"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.341693    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.341719    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.341837    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.341916    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.341926    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.342438    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 18:59:02.393799 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.342481    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.433143    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-89.ec2.internal\" not found" node="ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.442633    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.447510    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.448393    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.448433    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.448451    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.448487    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.448864    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.448906    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.448927    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.448966    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.451842    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.459656    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.504196 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.459675    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-89.ec2.internal\": node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.515678    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.549718    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal"]
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.549800    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.550837    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.550867    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.550880    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.554308    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.554448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.554489    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.560970    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.560991    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.561004    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.560970    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.561065    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.561078    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.562592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.562615    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.563300    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.563328    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.563342    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeHasSufficientPID"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.586270    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-89.ec2.internal\" not found" node="ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.607538 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.592156    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-89.ec2.internal\" not found" node="ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.613427 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.613406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b445083ebd080c8c5990df5245802d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal\" (UID: \"0b445083ebd080c8c5990df5245802d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.613490 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.613436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db49d6c0b18dfe371c7b893a3266857e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.ec2.internal\" (UID: \"db49d6c0b18dfe371c7b893a3266857e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.613490 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.613459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b445083ebd080c8c5990df5245802d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal\" (UID: \"0b445083ebd080c8c5990df5245802d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.616762 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.616728    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.713743 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.713707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b445083ebd080c8c5990df5245802d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal\" (UID: \"0b445083ebd080c8c5990df5245802d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.713893 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.713803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b445083ebd080c8c5990df5245802d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal\" (UID: \"0b445083ebd080c8c5990df5245802d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.713893 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.713833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db49d6c0b18dfe371c7b893a3266857e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.ec2.internal\" (UID: \"db49d6c0b18dfe371c7b893a3266857e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.713893 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.713773    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b445083ebd080c8c5990df5245802d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal\" (UID: \"0b445083ebd080c8c5990df5245802d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.713893 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.713870    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b445083ebd080c8c5990df5245802d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal\" (UID: \"0b445083ebd080c8c5990df5245802d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.714044 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.713906    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/db49d6c0b18dfe371c7b893a3266857e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-89.ec2.internal\" (UID: \"db49d6c0b18dfe371c7b893a3266857e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.717585 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.717346    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.817868 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.817777    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:02.894859 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.894815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.896074 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:02.896057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:02.918249 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:02.918220    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:03.018444 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:03.018393    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:03.032191 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.032162    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 18:59:03.032337 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.032319    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 18:59:03.118888 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:03.118811    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:03.143467 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.143442    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 18:59:03.204902 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.204860    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 18:54:02 +0000 UTC" deadline="2028-03-22 10:03:53.261536318 +0000 UTC"
Jul 08 18:59:03.204902 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.204900    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14943h4m50.056640427s"
Jul 08 18:59:03.219799 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:03.219771    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:03.320619 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:03.320587    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:03.421310 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:03.421211    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:03.521923 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:03.521887    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-89.ec2.internal\" not found"
Jul 08 18:59:03.563755 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:03.563709    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb49d6c0b18dfe371c7b893a3266857e.slice/crio-dd734594a218f41fe06870416cc88cfd5671444d72c73c7ab3e66fcfbb9e40af WatchSource:0}: Error finding container dd734594a218f41fe06870416cc88cfd5671444d72c73c7ab3e66fcfbb9e40af: Status 404 returned error can't find the container with id dd734594a218f41fe06870416cc88cfd5671444d72c73c7ab3e66fcfbb9e40af
Jul 08 18:59:03.563989 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:03.563970    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b445083ebd080c8c5990df5245802d6.slice/crio-ae14a49844852e83bc43e976163da7f0335f42fd4e44810f5f7a71114866fbc1 WatchSource:0}: Error finding container ae14a49844852e83bc43e976163da7f0335f42fd4e44810f5f7a71114866fbc1: Status 404 returned error can't find the container with id ae14a49844852e83bc43e976163da7f0335f42fd4e44810f5f7a71114866fbc1
Jul 08 18:59:03.567677 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.567661    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 18:59:03.595881 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.595862    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 18:59:03.605307 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.605286    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 18:59:03.705866 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.705802    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:03.716549 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.716532    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 18:59:03.718319 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.718307    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal"
Jul 08 18:59:03.729544 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:03.729526    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 18:59:04.088880 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.088554    2563 apiserver.go:52] "Watching apiserver"
Jul 08 18:59:04.098879 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.098796    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 18:59:04.099142 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.099119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn","openshift-cluster-node-tuning-operator/tuned-gr2pj","openshift-dns/node-resolver-n4x4s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal","openshift-multus/multus-additional-cni-plugins-5m9cq","openshift-multus/multus-zqgfd","openshift-multus/network-metrics-daemon-r6pt5","kube-system/global-pull-secret-syncer-mlmxj","kube-system/konnectivity-agent-dls7k","openshift-image-registry/node-ca-2ztqx","openshift-network-diagnostics/network-check-target-fwx64","openshift-network-operator/iptables-alerter-dtvvg","openshift-ovn-kubernetes/ovnkube-node-55wp8"]
Jul 08 18:59:04.101639 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.101607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.103801 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.103776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.105123 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.104907    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 18:59:04.105123 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.104910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 18:59:04.105502 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.105352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 18:59:04.105502 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.105360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvhkg\""
Jul 08 18:59:04.105502 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.105362    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 18:59:04.106769 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.105995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.106769 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.106751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 18:59:04.106904 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.106886    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8brz8\""
Jul 08 18:59:04.107024 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.107009    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 18:59:04.108868 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.108216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.108868 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.108308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 18:59:04.110546 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.110171    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 18:59:04.110546 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.110415    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.110546 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.110441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4ql8f\""
Jul 08 18:59:04.111240 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.111085    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 18:59:04.112762 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.112367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dxckj\""
Jul 08 18:59:04.112762 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.112589    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 18:59:04.112762 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.112596    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 18:59:04.112940 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.112787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.115169 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.114612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-997l4\""
Jul 08 18:59:04.115169 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.114718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 18:59:04.115169 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.114842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 18:59:04.115169 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.114933    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 18:59:04.115169 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.115025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 18:59:04.115442 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.115207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 18:59:04.115442 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.115406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 18:59:04.117522 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.115871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.117522 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.115946    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:04.117522 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.116362    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 18:59:04.117522 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.116552    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 18:59:04.118414 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.118360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.122695 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.119694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7w88f\""
Jul 08 18:59:04.122695 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.120942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.122695 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.122384    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 18:59:04.122981 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.122802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c15e4dd-dbca-407c-85b7-079bc169c852-cni-binary-copy\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.122981 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.122855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77bd5f3c-b7d3-4a92-b408-8366970336ef-serviceca\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.122981 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.122921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rszq\" (UniqueName: \"kubernetes.io/projected/77bd5f3c-b7d3-4a92-b408-8366970336ef-kube-api-access-8rszq\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.122981 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.122960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e81e8e-d568-4ffb-a26d-7975cf749b84-tmp\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123168 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.122992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-socket-dir-parent\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123168 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-daemon-config\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123168 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-modprobe-d\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123168 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-hostroot\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123168 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-kubernetes\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123168 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-tuned\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-system-cni-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-os-release\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-conf-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-multus-certs\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123278    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-etc-kubernetes\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-netns\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysconfig\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-host\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-cni-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-k8s-cni-cncf-io\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skk45\" (UniqueName: \"kubernetes.io/projected/7c15e4dd-dbca-407c-85b7-079bc169c852-kube-api-access-skk45\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s7vb8\""
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-sys\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcdd\" (UniqueName: \"kubernetes.io/projected/a9e81e8e-d568-4ffb-a26d-7975cf749b84-kube-api-access-tvcdd\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-cnibin\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysctl-d\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.123923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-cni-multus\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-kubelet\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.123980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-systemd\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.124016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-run\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.124055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-var-lib-kubelet\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.124084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77bd5f3c-b7d3-4a92-b408-8366970336ef-host\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.124103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-cni-bin\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.124125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysctl-conf\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.124308 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.124142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-lib-modules\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.124690 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.124573    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.126670 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.125959    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 18:59:04.126670 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.126138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87bt5\""
Jul 08 18:59:04.126670 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.126233    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 18:59:04.126670 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.126410    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 18:59:04.126909 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.126719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:04.126909 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.126793    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:04.128029 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.127921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 18:59:04.128923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.128903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 18:59:04.129399 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.129198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:04.129399 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.129264    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:04.129399 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.129281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tpkr8\""
Jul 08 18:59:04.206926 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.206904    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 18:59:04.207029 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.206976    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 18:59:04.224813 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.224788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-cni-netd\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.224935 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.224827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-system-cni-dir\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.224935 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.224852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.224935 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.224877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovnkube-config\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.224935 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.224907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-kubernetes\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.225129 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.224947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-tuned\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.225129 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.224981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-os-release\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225129 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-conf-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225129 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225025    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-kubernetes\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.225129 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-kubelet\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.225129 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225086    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-conf-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225129 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-cni-bin\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225135    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-os-release\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-netns\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225199    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-netns\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysconfig\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-systemd\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225240    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysconfig\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667pj\" (UniqueName: \"kubernetes.io/projected/43f058dd-d85e-4c3c-9212-3e7095606619-kube-api-access-667pj\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225258    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kwl\" (UniqueName: \"kubernetes.io/projected/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-kube-api-access-w4kwl\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-sys\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-cnibin\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225377    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-sys\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovn-node-metrics-cert\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225453    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-cnibin\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.225515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-device-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysctl-d\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-kubelet\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-tmp-dir\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-systemd\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225677    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-kubelet\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225690    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysctl-d\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-run\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225753    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-systemd\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-var-lib-kubelet\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77bd5f3c-b7d3-4a92-b408-8366970336ef-host\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225828    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-var-lib-kubelet\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225843    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-run\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-systemd-units\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225882    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77bd5f3c-b7d3-4a92-b408-8366970336ef-host\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-os-release\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.226286 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-dbus\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.225975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-socket-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-lib-modules\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c15e4dd-dbca-407c-85b7-079bc169c852-cni-binary-copy\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rszq\" (UniqueName: \"kubernetes.io/projected/77bd5f3c-b7d3-4a92-b408-8366970336ef-kube-api-access-8rszq\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-log-socket\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226142    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-lib-modules\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-run-ovn-kubernetes\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-env-overrides\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df0230e9-590c-4a18-9dde-52eaa9fdb8ce-konnectivity-ca\") pod \"konnectivity-agent-dls7k\" (UID: \"df0230e9-590c-4a18-9dde-52eaa9fdb8ce\") " pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-registration-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e81e8e-d568-4ffb-a26d-7975cf749b84-tmp\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-socket-dir-parent\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-daemon-config\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226399    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-socket-dir-parent\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-var-lib-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df0230e9-590c-4a18-9dde-52eaa9fdb8ce-agent-certs\") pod \"konnectivity-agent-dls7k\" (UID: \"df0230e9-590c-4a18-9dde-52eaa9fdb8ce\") " pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.227093 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-modprobe-d\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-run-netns\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrdg\" (UniqueName: \"kubernetes.io/projected/b9539e97-e9f8-4368-8224-f1fdb0d667f2-kube-api-access-mmrdg\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-system-cni-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226683    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-modprobe-d\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-multus-certs\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226758    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-multus-certs\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226758    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c15e4dd-dbca-407c-85b7-079bc169c852-cni-binary-copy\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-etc-kubernetes\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226830    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-system-cni-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-etc-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226868    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-etc-kubernetes\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226874    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-daemon-config\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgs6\" (UniqueName: \"kubernetes.io/projected/304f1fb3-642c-4933-bc87-3db8e0afbefc-kube-api-access-4xgs6\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-cni-binary-copy\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-host-slash\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.226984    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:04.227925 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-host\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-cni-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-k8s-cni-cncf-io\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skk45\" (UniqueName: \"kubernetes.io/projected/7c15e4dd-dbca-407c-85b7-079bc169c852-kube-api-access-skk45\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227075    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-host\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-hosts-file\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-kubelet-config\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-run-k8s-cni-cncf-io\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcdd\" (UniqueName: \"kubernetes.io/projected/a9e81e8e-d568-4ffb-a26d-7975cf749b84-kube-api-access-tvcdd\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227151    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-multus-cni-dir\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-cnibin\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-cni-multus\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227341    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-cni-multus\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovnkube-script-lib\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwft9\" (UniqueName: \"kubernetes.io/projected/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-kube-api-access-jwft9\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.228701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-sys-fs\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-slash\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-cni-bin\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysctl-conf\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77bd5f3c-b7d3-4a92-b408-8366970336ef-serviceca\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227579    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-host-var-lib-cni-bin\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227738    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-sysctl-conf\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228026    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/77bd5f3c-b7d3-4a92-b408-8366970336ef-serviceca\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.227598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-iptables-alerter-script\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstlt\" (UniqueName: \"kubernetes.io/projected/daca3b10-16da-4559-8498-a0bfec82aad8-kube-api-access-rstlt\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-node-log\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-hostroot\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-ovn\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228324    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c15e4dd-dbca-407c-85b7-079bc169c852-hostroot\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.229252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228855    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a9e81e8e-d568-4ffb-a26d-7975cf749b84-etc-tuned\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.229783 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.228943    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9e81e8e-d568-4ffb-a26d-7975cf749b84-tmp\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.236121 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.236094    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 18:59:04.247295 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.247265    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skk45\" (UniqueName: \"kubernetes.io/projected/7c15e4dd-dbca-407c-85b7-079bc169c852-kube-api-access-skk45\") pod \"multus-zqgfd\" (UID: \"7c15e4dd-dbca-407c-85b7-079bc169c852\") " pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.248722 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.248706    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcdd\" (UniqueName: \"kubernetes.io/projected/a9e81e8e-d568-4ffb-a26d-7975cf749b84-kube-api-access-tvcdd\") pod \"tuned-gr2pj\" (UID: \"a9e81e8e-d568-4ffb-a26d-7975cf749b84\") " pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.254765 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.254743    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rszq\" (UniqueName: \"kubernetes.io/projected/77bd5f3c-b7d3-4a92-b408-8366970336ef-kube-api-access-8rszq\") pod \"node-ca-2ztqx\" (UID: \"77bd5f3c-b7d3-4a92-b408-8366970336ef\") " pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.277542 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.277518    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltqp8"
Jul 08 18:59:04.289345 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.289321    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltqp8"
Jul 08 18:59:04.328562 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.328700 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-iptables-alerter-script\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.328700 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstlt\" (UniqueName: \"kubernetes.io/projected/daca3b10-16da-4559-8498-a0bfec82aad8-kube-api-access-rstlt\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:04.328700 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:04.328700 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-node-log\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328700 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-ovn\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-cni-netd\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-system-cni-dir\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovnkube-config\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-kubelet\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-cni-bin\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.328896 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-systemd\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667pj\" (UniqueName: \"kubernetes.io/projected/43f058dd-d85e-4c3c-9212-3e7095606619-kube-api-access-667pj\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kwl\" (UniqueName: \"kubernetes.io/projected/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-kube-api-access-w4kwl\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovn-node-metrics-cert\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.328994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-device-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-tmp-dir\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-systemd-units\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-os-release\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-dbus\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329170    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-socket-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-log-socket\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329251 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-run-ovn-kubernetes\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-env-overrides\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329284    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-socket-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df0230e9-590c-4a18-9dde-52eaa9fdb8ce-konnectivity-ca\") pod \"konnectivity-agent-dls7k\" (UID: \"df0230e9-590c-4a18-9dde-52eaa9fdb8ce\") " pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-registration-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-var-lib-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df0230e9-590c-4a18-9dde-52eaa9fdb8ce-agent-certs\") pod \"konnectivity-agent-dls7k\" (UID: \"df0230e9-590c-4a18-9dde-52eaa9fdb8ce\") " pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-run-netns\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrdg\" (UniqueName: \"kubernetes.io/projected/b9539e97-e9f8-4368-8224-f1fdb0d667f2-kube-api-access-mmrdg\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-etc-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgs6\" (UniqueName: \"kubernetes.io/projected/304f1fb3-642c-4933-bc87-3db8e0afbefc-kube-api-access-4xgs6\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-cni-binary-copy\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-host-slash\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-hosts-file\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-kubelet-config\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-cnibin\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.329865 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovnkube-script-lib\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwft9\" (UniqueName: \"kubernetes.io/projected/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-kube-api-access-jwft9\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329808    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-iptables-alerter-script\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-sys-fs\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329870    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-run-ovn-kubernetes\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329920    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-host-slash\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329926    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-node-log\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329948    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-run-netns\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329955    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.329959    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.329985    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-etc-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330024    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-ovn\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330071    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-hosts-file\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330216    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-system-cni-dir\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330259    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.330578 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330437    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330471    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-cni-binary-copy\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330486    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-kubelet\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330535    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-cni-netd\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330583    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330609    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-env-overrides\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330683    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-cni-bin\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330723    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-var-lib-openvswitch\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330748    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovnkube-config\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330780    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-registration-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330800    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df0230e9-590c-4a18-9dde-52eaa9fdb8ce-konnectivity-ca\") pod \"konnectivity-agent-dls7k\" (UID: \"df0230e9-590c-4a18-9dde-52eaa9fdb8ce\") " pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330822    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-run-systemd\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330858    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-os-release\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330874    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-dbus\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.330909    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330920    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-systemd-units\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330985    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-kubelet-config\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331026    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43f058dd-d85e-4c3c-9212-3e7095606619-cnibin\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.331281 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-slash\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.331128    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:04.83110776 +0000 UTC m=+3.475128469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.331176    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret podName:e43d8623-63ae-4e5b-92e7-fb1db07f2cec nodeName:}" failed. No retries permitted until 2026-07-08 18:59:04.831164614 +0000 UTC m=+3.475185328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret") pod "global-pull-secret-syncer-mlmxj" (UID: "e43d8623-63ae-4e5b-92e7-fb1db07f2cec") : object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331209    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-sys-fs\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331211    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-host-slash\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331238    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/304f1fb3-642c-4933-bc87-3db8e0afbefc-log-socket\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331274    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.330816    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9539e97-e9f8-4368-8224-f1fdb0d667f2-device-dir\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331446    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovnkube-script-lib\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331480    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43f058dd-d85e-4c3c-9212-3e7095606619-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.332133 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.331554    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-tmp-dir\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.332845 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.332821    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df0230e9-590c-4a18-9dde-52eaa9fdb8ce-agent-certs\") pod \"konnectivity-agent-dls7k\" (UID: \"df0230e9-590c-4a18-9dde-52eaa9fdb8ce\") " pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.333744 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.333723    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/304f1fb3-642c-4933-bc87-3db8e0afbefc-ovn-node-metrics-cert\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.343001 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.342954    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 18:59:04.343001 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.342977    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 18:59:04.343001 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.342990    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mzmld for pod openshift-network-diagnostics/network-check-target-fwx64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:04.343189 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.343040    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld podName:252c1061-ec85-4b07-9a88-d0a796e1eac4 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:04.843024691 +0000 UTC m=+3.487045414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mzmld" (UniqueName: "kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld") pod "network-check-target-fwx64" (UID: "252c1061-ec85-4b07-9a88-d0a796e1eac4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:04.346699 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.346654    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kwl\" (UniqueName: \"kubernetes.io/projected/4fea7aea-3311-4b3a-bbf9-6a0c2585aec6-kube-api-access-w4kwl\") pod \"node-resolver-n4x4s\" (UID: \"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6\") " pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.351665 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.351643    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-667pj\" (UniqueName: \"kubernetes.io/projected/43f058dd-d85e-4c3c-9212-3e7095606619-kube-api-access-667pj\") pod \"multus-additional-cni-plugins-5m9cq\" (UID: \"43f058dd-d85e-4c3c-9212-3e7095606619\") " pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.353282 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.353243    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstlt\" (UniqueName: \"kubernetes.io/projected/daca3b10-16da-4559-8498-a0bfec82aad8-kube-api-access-rstlt\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:04.355894 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.355845    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgs6\" (UniqueName: \"kubernetes.io/projected/304f1fb3-642c-4933-bc87-3db8e0afbefc-kube-api-access-4xgs6\") pod \"ovnkube-node-55wp8\" (UID: \"304f1fb3-642c-4933-bc87-3db8e0afbefc\") " pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.357382 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.357360    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwft9\" (UniqueName: \"kubernetes.io/projected/cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c-kube-api-access-jwft9\") pod \"iptables-alerter-dtvvg\" (UID: \"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c\") " pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.362560 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.362541    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrdg\" (UniqueName: \"kubernetes.io/projected/b9539e97-e9f8-4368-8224-f1fdb0d667f2-kube-api-access-mmrdg\") pod \"aws-ebs-csi-driver-node-mc5zn\" (UID: \"b9539e97-e9f8-4368-8224-f1fdb0d667f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.392807 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.392783    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 18:59:04.417731 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.417708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zqgfd"
Jul 08 18:59:04.427247 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.427223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2ztqx"
Jul 08 18:59:04.429672 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.429646    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c15e4dd_dbca_407c_85b7_079bc169c852.slice/crio-06344de8c3b5def4ba7e08c4bc5051b6fa01498bf38f187b260d8ff5d8287e99 WatchSource:0}: Error finding container 06344de8c3b5def4ba7e08c4bc5051b6fa01498bf38f187b260d8ff5d8287e99: Status 404 returned error can't find the container with id 06344de8c3b5def4ba7e08c4bc5051b6fa01498bf38f187b260d8ff5d8287e99
Jul 08 18:59:04.436779 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.436756    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bd5f3c_b7d3_4a92_b408_8366970336ef.slice/crio-2935cfceffc4561dfbf4d24648d9ddcfb7b0b0f3f38425e79896bba6bd0735b7 WatchSource:0}: Error finding container 2935cfceffc4561dfbf4d24648d9ddcfb7b0b0f3f38425e79896bba6bd0735b7: Status 404 returned error can't find the container with id 2935cfceffc4561dfbf4d24648d9ddcfb7b0b0f3f38425e79896bba6bd0735b7
Jul 08 18:59:04.438531 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.438510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gr2pj"
Jul 08 18:59:04.444558 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.444539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n4x4s"
Jul 08 18:59:04.447740 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.447614    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e81e8e_d568_4ffb_a26d_7975cf749b84.slice/crio-6b9783458e19f91173c5968e968eee769cf57dd4763227979ad5e4c8d5c56e69 WatchSource:0}: Error finding container 6b9783458e19f91173c5968e968eee769cf57dd4763227979ad5e4c8d5c56e69: Status 404 returned error can't find the container with id 6b9783458e19f91173c5968e968eee769cf57dd4763227979ad5e4c8d5c56e69
Jul 08 18:59:04.453687 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.453668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:04.455370 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.455345    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fea7aea_3311_4b3a_bbf9_6a0c2585aec6.slice/crio-332679bc9e56f75599fb1e4c50206267e24b1cdf532fc9d8a143ca71853c7b78 WatchSource:0}: Error finding container 332679bc9e56f75599fb1e4c50206267e24b1cdf532fc9d8a143ca71853c7b78: Status 404 returned error can't find the container with id 332679bc9e56f75599fb1e4c50206267e24b1cdf532fc9d8a143ca71853c7b78
Jul 08 18:59:04.455814 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.455770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zqgfd" event={"ID":"7c15e4dd-dbca-407c-85b7-079bc169c852","Type":"ContainerStarted","Data":"06344de8c3b5def4ba7e08c4bc5051b6fa01498bf38f187b260d8ff5d8287e99"}
Jul 08 18:59:04.458717 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.458682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal" event={"ID":"0b445083ebd080c8c5990df5245802d6","Type":"ContainerStarted","Data":"ae14a49844852e83bc43e976163da7f0335f42fd4e44810f5f7a71114866fbc1"}
Jul 08 18:59:04.461228 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.461070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5m9cq"
Jul 08 18:59:04.461671 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.461595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal" event={"ID":"db49d6c0b18dfe371c7b893a3266857e","Type":"ContainerStarted","Data":"dd734594a218f41fe06870416cc88cfd5671444d72c73c7ab3e66fcfbb9e40af"}
Jul 08 18:59:04.463260 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.462852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gr2pj" event={"ID":"a9e81e8e-d568-4ffb-a26d-7975cf749b84","Type":"ContainerStarted","Data":"6b9783458e19f91173c5968e968eee769cf57dd4763227979ad5e4c8d5c56e69"}
Jul 08 18:59:04.464803 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.464783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2ztqx" event={"ID":"77bd5f3c-b7d3-4a92-b408-8366970336ef","Type":"ContainerStarted","Data":"2935cfceffc4561dfbf4d24648d9ddcfb7b0b0f3f38425e79896bba6bd0735b7"}
Jul 08 18:59:04.465352 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.465332    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304f1fb3_642c_4933_bc87_3db8e0afbefc.slice/crio-38c6f0b464f1379de16dae5cd3316ae2ef84fd4ada581561f673750fb19463a4 WatchSource:0}: Error finding container 38c6f0b464f1379de16dae5cd3316ae2ef84fd4ada581561f673750fb19463a4: Status 404 returned error can't find the container with id 38c6f0b464f1379de16dae5cd3316ae2ef84fd4ada581561f673750fb19463a4
Jul 08 18:59:04.470859 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.470834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dtvvg"
Jul 08 18:59:04.470959 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.470936    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f058dd_d85e_4c3c_9212_3e7095606619.slice/crio-da7db6ba8b633f584a319977db798f2999978c190459327adb9c78a406d5ce98 WatchSource:0}: Error finding container da7db6ba8b633f584a319977db798f2999978c190459327adb9c78a406d5ce98: Status 404 returned error can't find the container with id da7db6ba8b633f584a319977db798f2999978c190459327adb9c78a406d5ce98
Jul 08 18:59:04.477500 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.477485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn"
Jul 08 18:59:04.482081 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.482060    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf679dfa_17a7_4dbd_97b2_6d6f2ac18e5c.slice/crio-78729ef15dca29f099048003658cdcbcf386b353af019a82ebdf44b873f6f735 WatchSource:0}: Error finding container 78729ef15dca29f099048003658cdcbcf386b353af019a82ebdf44b873f6f735: Status 404 returned error can't find the container with id 78729ef15dca29f099048003658cdcbcf386b353af019a82ebdf44b873f6f735
Jul 08 18:59:04.484508 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.484141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:04.496094 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:04.496066    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0230e9_590c_4a18_9dde_52eaa9fdb8ce.slice/crio-7d1092c05128ded65d8890039e4ff38ea2c3dd9a9afb5d4795c5ce28d028aeb6 WatchSource:0}: Error finding container 7d1092c05128ded65d8890039e4ff38ea2c3dd9a9afb5d4795c5ce28d028aeb6: Status 404 returned error can't find the container with id 7d1092c05128ded65d8890039e4ff38ea2c3dd9a9afb5d4795c5ce28d028aeb6
Jul 08 18:59:04.833052 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.832966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:04.833052 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.833014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:04.833247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.833110    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:04.833247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.833162    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:04.833247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.833190    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:05.833170724 +0000 UTC m=+4.477191434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:04.833247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.833222    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret podName:e43d8623-63ae-4e5b-92e7-fb1db07f2cec nodeName:}" failed. No retries permitted until 2026-07-08 18:59:05.833204669 +0000 UTC m=+4.477225400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret") pod "global-pull-secret-syncer-mlmxj" (UID: "e43d8623-63ae-4e5b-92e7-fb1db07f2cec") : object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:04.933965 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:04.933929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:04.934142 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.934115    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 18:59:04.934142 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.934140    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 18:59:04.934252 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.934153    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mzmld for pod openshift-network-diagnostics/network-check-target-fwx64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:04.934252 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:04.934219    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld podName:252c1061-ec85-4b07-9a88-d0a796e1eac4 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:05.93420024 +0000 UTC m=+4.578220975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzmld" (UniqueName: "kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld") pod "network-check-target-fwx64" (UID: "252c1061-ec85-4b07-9a88-d0a796e1eac4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:05.293746 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.293676    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:54:04 +0000 UTC" deadline="2028-02-14 09:24:19.520643589 +0000 UTC"
Jul 08 18:59:05.293746 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.293705    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14054h25m14.22694207s"
Jul 08 18:59:05.450006 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.449957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:05.450134 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.450097    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:05.450552 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.450532    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:05.450935 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.450654    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:05.473507 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.473403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"38c6f0b464f1379de16dae5cd3316ae2ef84fd4ada581561f673750fb19463a4"}
Jul 08 18:59:05.480317 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.480248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n4x4s" event={"ID":"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6","Type":"ContainerStarted","Data":"332679bc9e56f75599fb1e4c50206267e24b1cdf532fc9d8a143ca71853c7b78"}
Jul 08 18:59:05.482647 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.482553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn" event={"ID":"b9539e97-e9f8-4368-8224-f1fdb0d667f2","Type":"ContainerStarted","Data":"7ca8d9162586bfe9c2d9aaf5bfede4068daa363f2b312661bf91a820bb8915bf"}
Jul 08 18:59:05.485220 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.485144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dls7k" event={"ID":"df0230e9-590c-4a18-9dde-52eaa9fdb8ce","Type":"ContainerStarted","Data":"7d1092c05128ded65d8890039e4ff38ea2c3dd9a9afb5d4795c5ce28d028aeb6"}
Jul 08 18:59:05.489680 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.489589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dtvvg" event={"ID":"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c","Type":"ContainerStarted","Data":"78729ef15dca29f099048003658cdcbcf386b353af019a82ebdf44b873f6f735"}
Jul 08 18:59:05.498549 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.498521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerStarted","Data":"da7db6ba8b633f584a319977db798f2999978c190459327adb9c78a406d5ce98"}
Jul 08 18:59:05.842903 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.842859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:05.843084 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.842964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:05.843156 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.843086    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:05.843156 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.843145    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:07.843126171 +0000 UTC m=+6.487146899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:05.843552 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.843524    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:05.843656 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.843573    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret podName:e43d8623-63ae-4e5b-92e7-fb1db07f2cec nodeName:}" failed. No retries permitted until 2026-07-08 18:59:07.843557333 +0000 UTC m=+6.487578055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret") pod "global-pull-secret-syncer-mlmxj" (UID: "e43d8623-63ae-4e5b-92e7-fb1db07f2cec") : object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:05.943452 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:05.943418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:05.943673 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.943647    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 18:59:05.943749 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.943679    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 18:59:05.943749 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.943694    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mzmld for pod openshift-network-diagnostics/network-check-target-fwx64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:05.943856 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:05.943760    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld podName:252c1061-ec85-4b07-9a88-d0a796e1eac4 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:07.943737899 +0000 UTC m=+6.587758630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzmld" (UniqueName: "kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld") pod "network-check-target-fwx64" (UID: "252c1061-ec85-4b07-9a88-d0a796e1eac4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:06.294641 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:06.294415    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:54:04 +0000 UTC" deadline="2028-03-30 14:37:35.131363886 +0000 UTC"
Jul 08 18:59:06.294641 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:06.294641    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15139h38m28.836728335s"
Jul 08 18:59:06.450742 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:06.450136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:06.450742 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:06.450266    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:06.522719 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:06.522682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal" event={"ID":"db49d6c0b18dfe371c7b893a3266857e","Type":"ContainerStarted","Data":"75c7a51286b05f9b669c116d6a3cbee8b6b72545069e11579cd77272eba802d8"}
Jul 08 18:59:07.449152 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.449103    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:07.449609 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.449233    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:07.449609 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.449600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:07.449793 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.449717    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:07.534053 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.534015    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b445083ebd080c8c5990df5245802d6" containerID="d83b44caab7901760631c0c437e83ba15a80e4c6607996171ad76cb709f46648" exitCode=0
Jul 08 18:59:07.534951 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.534924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal" event={"ID":"0b445083ebd080c8c5990df5245802d6","Type":"ContainerDied","Data":"d83b44caab7901760631c0c437e83ba15a80e4c6607996171ad76cb709f46648"}
Jul 08 18:59:07.561285 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.560336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-89.ec2.internal" podStartSLOduration=4.560315958 podStartE2EDuration="4.560315958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:06.544782533 +0000 UTC m=+5.188803264" watchObservedRunningTime="2026-07-08 18:59:07.560315958 +0000 UTC m=+6.204336689"
Jul 08 18:59:07.859441 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.858644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:07.859441 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.858701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:07.859441 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.858857    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:07.859441 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.858918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret podName:e43d8623-63ae-4e5b-92e7-fb1db07f2cec nodeName:}" failed. No retries permitted until 2026-07-08 18:59:11.85889754 +0000 UTC m=+10.502918265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret") pod "global-pull-secret-syncer-mlmxj" (UID: "e43d8623-63ae-4e5b-92e7-fb1db07f2cec") : object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:07.859441 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.859312    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:07.859441 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.859365    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:11.859348769 +0000 UTC m=+10.503369477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:07.960476 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:07.959735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:07.960476 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.959944    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 18:59:07.960476 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.959965    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 18:59:07.960476 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.959977    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mzmld for pod openshift-network-diagnostics/network-check-target-fwx64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:07.960476 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:07.960035    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld podName:252c1061-ec85-4b07-9a88-d0a796e1eac4 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:11.960017701 +0000 UTC m=+10.604038417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzmld" (UniqueName: "kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld") pod "network-check-target-fwx64" (UID: "252c1061-ec85-4b07-9a88-d0a796e1eac4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:08.450318 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:08.450248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:08.450797 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:08.450383    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:09.450402 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:09.449821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:09.450402 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:09.449871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:09.450402 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:09.449966    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:09.450402 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:09.450365    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:10.449585 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:10.449359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:10.449585 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:10.449470    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:11.449734 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:11.449703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:11.450125 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:11.449703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:11.450125 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.449844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:11.450125 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.449908    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:11.895032 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:11.894911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:11.895032 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:11.894969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:11.895247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.895056    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:11.895247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.895068    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:11.895247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.895137    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:19.895114556 +0000 UTC m=+18.539135281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:11.895247 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.895159    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret podName:e43d8623-63ae-4e5b-92e7-fb1db07f2cec nodeName:}" failed. No retries permitted until 2026-07-08 18:59:19.895149885 +0000 UTC m=+18.539170605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret") pod "global-pull-secret-syncer-mlmxj" (UID: "e43d8623-63ae-4e5b-92e7-fb1db07f2cec") : object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:11.995351 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:11.995301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:11.995517 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.995483    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 18:59:11.995517 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.995511    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 18:59:11.995656 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.995525    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mzmld for pod openshift-network-diagnostics/network-check-target-fwx64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:11.995656 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:11.995589    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld podName:252c1061-ec85-4b07-9a88-d0a796e1eac4 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:19.995569554 +0000 UTC m=+18.639590292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzmld" (UniqueName: "kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld") pod "network-check-target-fwx64" (UID: "252c1061-ec85-4b07-9a88-d0a796e1eac4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:12.450589 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:12.450559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:12.451055 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:12.450669    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:13.449807 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:13.449772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:13.450012 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:13.449914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:13.450424 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:13.450293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:13.450424 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:13.450382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:14.450220 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:14.450184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:14.450684 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:14.450321    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:15.449859 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:15.449825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:15.449859 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:15.449832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:15.450079 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:15.449994    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:15.450141 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:15.450122    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:16.450096 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:16.450032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:16.450570 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:16.450167    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:17.450039 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:17.450005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:17.450211 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:17.450010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:17.450211 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:17.450165    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:17.450565 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:17.450243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:18.449541 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:18.449513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:18.449723 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:18.449634    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:19.449202 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:19.449139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:19.449202 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:19.449178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:19.449798 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:19.449262    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:19.449798 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:19.449347    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:19.952259 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:19.952226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:19.952448 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:19.952265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:19.952448 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:19.952388    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:19.952448 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:19.952446    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret podName:e43d8623-63ae-4e5b-92e7-fb1db07f2cec nodeName:}" failed. No retries permitted until 2026-07-08 18:59:35.952431667 +0000 UTC m=+34.596452376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret") pod "global-pull-secret-syncer-mlmxj" (UID: "e43d8623-63ae-4e5b-92e7-fb1db07f2cec") : object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:19.952638 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:19.952388    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:19.952638 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:19.952515    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:35.952501935 +0000 UTC m=+34.596522647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:20.052610 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:20.052568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:20.052785 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:20.052724    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 18:59:20.052785 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:20.052744    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 18:59:20.052785 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:20.052755    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mzmld for pod openshift-network-diagnostics/network-check-target-fwx64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:20.052883 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:20.052818    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld podName:252c1061-ec85-4b07-9a88-d0a796e1eac4 nodeName:}" failed. No retries permitted until 2026-07-08 18:59:36.05279878 +0000 UTC m=+34.696819496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzmld" (UniqueName: "kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld") pod "network-check-target-fwx64" (UID: "252c1061-ec85-4b07-9a88-d0a796e1eac4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:20.450127 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:20.450039    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:20.450652 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:20.450177    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:21.449329 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:21.449288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:21.449329 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:21.449328    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:21.449562 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:21.449421    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:21.449562 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:21.449553    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:22.450223 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:22.450188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:22.450653 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:22.450305    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:23.449851 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:23.449821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:23.449958 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:23.449939    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:23.450016 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:23.450000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:23.450101 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:23.450083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:24.449538 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.449174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:24.450050 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:24.449662    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:24.568012 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.567981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zqgfd" event={"ID":"7c15e4dd-dbca-407c-85b7-079bc169c852","Type":"ContainerStarted","Data":"0d9e15f47c2d72e6dbbb9ad9a6ab2d432e5fe8a219ab2e9705548c547dd43848"}
Jul 08 18:59:24.570073 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.570048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal" event={"ID":"0b445083ebd080c8c5990df5245802d6","Type":"ContainerStarted","Data":"4c8f4d5b89d0cc198eff69155fd05fc594e816dcb9b290cb4b06c1966a624b1f"}
Jul 08 18:59:24.571941 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.571738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn" event={"ID":"b9539e97-e9f8-4368-8224-f1fdb0d667f2","Type":"ContainerStarted","Data":"07af030915a84eb7f9924b80e58598d8828eb88e5adbef876e8de9978251174b"}
Jul 08 18:59:24.573397 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.573356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gr2pj" event={"ID":"a9e81e8e-d568-4ffb-a26d-7975cf749b84","Type":"ContainerStarted","Data":"2cc698ff1e7b706f21894722b43c4b919f2506c80e2923171f02d58de2bf8a56"}
Jul 08 18:59:24.574929 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.574907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2ztqx" event={"ID":"77bd5f3c-b7d3-4a92-b408-8366970336ef","Type":"ContainerStarted","Data":"d73b16e62237b46e8ff319c24ad89e24b4da8c808afc64b56049ed9110350163"}
Jul 08 18:59:24.576553 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.576513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dls7k" event={"ID":"df0230e9-590c-4a18-9dde-52eaa9fdb8ce","Type":"ContainerStarted","Data":"2c140263f6d553b7f1b73d4f3ecce79b57e2e6e8445f8f7666eb3034c974f5d4"}
Jul 08 18:59:24.578171 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.578133    2563 generic.go:356] "Generic (PLEG): container finished" podID="43f058dd-d85e-4c3c-9212-3e7095606619" containerID="7ed066f79d0b4e9a74de9e3893293c39fddf280a003d3523f59224b8492d9ae0" exitCode=0
Jul 08 18:59:24.578264 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.578215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerDied","Data":"7ed066f79d0b4e9a74de9e3893293c39fddf280a003d3523f59224b8492d9ae0"}
Jul 08 18:59:24.581127 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.581109    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 18:59:24.581471 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.581446    2563 generic.go:356] "Generic (PLEG): container finished" podID="304f1fb3-642c-4933-bc87-3db8e0afbefc" containerID="2b02339573437df431e81bd7cbc50d5c6caeb5b612a59c81f1cbeaf24ce0ba19" exitCode=1
Jul 08 18:59:24.581579 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.581479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"566faf3e18efe310d02a8a142f9989e9e880f3ba862335ca14881486ea5ddb86"}
Jul 08 18:59:24.581579 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.581544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"c7930f509e89d834a02ac8d464578f0edd4d924674c5b6a564d2c80011dd41fe"}
Jul 08 18:59:24.581720 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.581596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"4809d27e2b34c281a9605781b09aa0714c90bd2b3f15ec1c355f4d71c6adb457"}
Jul 08 18:59:24.581720 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.581658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerDied","Data":"2b02339573437df431e81bd7cbc50d5c6caeb5b612a59c81f1cbeaf24ce0ba19"}
Jul 08 18:59:24.581817 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.581720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"1d21d0b2bc29a139511e7d20b3292d564ba8399804b2f513200b90a3978fab10"}
Jul 08 18:59:24.583080 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.583062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n4x4s" event={"ID":"4fea7aea-3311-4b3a-bbf9-6a0c2585aec6","Type":"ContainerStarted","Data":"e6d22d4bbc0ec5743d9fcc6db86f3c1197c0a419b9a38a2209e9505c00246e7f"}
Jul 08 18:59:24.590694 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.590655    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zqgfd" podStartSLOduration=3.533595617 podStartE2EDuration="22.590639319s" totalImagesPullingTime="19.057043702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:24.590150748 +0000 UTC m=+23.234171479" watchObservedRunningTime="2026-07-08 18:59:24.590639319 +0000 UTC m=+23.234660046"
Jul 08 18:59:24.606646 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.606357    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dls7k" podStartSLOduration=3.671657453 podStartE2EDuration="22.6063388s" totalImagesPullingTime="18.934681347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:24.605457533 +0000 UTC m=+23.249478251" watchObservedRunningTime="2026-07-08 18:59:24.6063388 +0000 UTC m=+23.250359531"
Jul 08 18:59:24.624425 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.624375    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gr2pj" podStartSLOduration=3.598481812 podStartE2EDuration="22.624358865s" totalImagesPullingTime="19.025877053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:24.623763584 +0000 UTC m=+23.267784314" watchObservedRunningTime="2026-07-08 18:59:24.624358865 +0000 UTC m=+23.268379596"
Jul 08 18:59:24.642555 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.642514    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n4x4s" podStartSLOduration=4.006689192 podStartE2EDuration="22.642502367s" totalImagesPullingTime="18.635813175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:24.642035265 +0000 UTC m=+23.286055995" watchObservedRunningTime="2026-07-08 18:59:24.642502367 +0000 UTC m=+23.286523096"
Jul 08 18:59:24.671073 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.671035    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-89.ec2.internal" podStartSLOduration=21.671024657 podStartE2EDuration="21.671024657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:24.670526277 +0000 UTC m=+23.314547008" watchObservedRunningTime="2026-07-08 18:59:24.671024657 +0000 UTC m=+23.315045386"
Jul 08 18:59:24.698008 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:24.697964    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2ztqx" podStartSLOduration=3.703916686 podStartE2EDuration="22.697952265s" totalImagesPullingTime="18.994035579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:24.69760638 +0000 UTC m=+23.341627110" watchObservedRunningTime="2026-07-08 18:59:24.697952265 +0000 UTC m=+23.341972994"
Jul 08 18:59:25.449387 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:25.449351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:25.449519 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:25.449481    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:25.449923 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:25.449902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:25.450276 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:25.450012    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:25.528289 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:25.528250    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 18:59:25.587254 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:25.587219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn" event={"ID":"b9539e97-e9f8-4368-8224-f1fdb0d667f2","Type":"ContainerStarted","Data":"2ab13af616c07ab57dca776245ad765bb7271165851b038d60872b7d6001c887"}
Jul 08 18:59:25.588615 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:25.588590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dtvvg" event={"ID":"cf679dfa-17a7-4dbd-97b2-6d6f2ac18e5c","Type":"ContainerStarted","Data":"0de9c485d1079677ac705c82358503d84f9e694bbf175e526bbec441112b12ca"}
Jul 08 18:59:25.591357 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:25.591341    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 18:59:25.591727 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:25.591701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"6a83fa2f54882c32fb176bf05a14c6b4d8a93901893a639e8f206193dc500011"}
Jul 08 18:59:26.354742 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:26.354641    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T18:59:25.528268854Z","UUID":"fb6dbdab-2f15-48ac-9cad-07b1b12e2d02","Handler":null,"Name":"","Endpoint":""}
Jul 08 18:59:26.356110 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:26.356083    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 18:59:26.356244 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:26.356127    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 18:59:26.449874 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:26.449841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:26.450063 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:26.449978    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:27.450147 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.449871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:27.450767 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.449898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:27.450767 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:27.450181    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:27.450767 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:27.450302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:27.579145 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.579107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:27.580213 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.580195    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:27.598106 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.598067    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dtvvg" podStartSLOduration=6.609481847 podStartE2EDuration="25.598049984s" totalImagesPullingTime="18.988568137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:25.607806079 +0000 UTC m=+24.251826808" watchObservedRunningTime="2026-07-08 18:59:27.598049984 +0000 UTC m=+26.242070715"
Jul 08 18:59:27.599178 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.599159    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 18:59:27.599540 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.599519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"100bc8b76301905a2d56b545ae68e9e37bca59449d9b6a6e8e41b4b80ec4dc3c"}
Jul 08 18:59:27.601494 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.601472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn" event={"ID":"b9539e97-e9f8-4368-8224-f1fdb0d667f2","Type":"ContainerStarted","Data":"bc6bba3fcbbf162d15ce5b2d93d7e1c9317f356b9199c1b22cbeec7e2fb31fe0"}
Jul 08 18:59:27.622653 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:27.622585    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mc5zn" podStartSLOduration=3.846252033 podStartE2EDuration="25.622569107s" totalImagesPullingTime="21.776317074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:27.622131689 +0000 UTC m=+26.266152419" watchObservedRunningTime="2026-07-08 18:59:27.622569107 +0000 UTC m=+26.266589837"
Jul 08 18:59:28.449508 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:28.449475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:28.449690 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:28.449580    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:29.449780 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.449543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:29.450456 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.449582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:29.450456 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:29.449889    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:29.450456 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:29.449929    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:29.609585 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.609553    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 18:59:29.609976 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.609947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"96288b7732251e238386408ba4b500d380d7a7d031be87b7c39d0c65e2eafd8b"}
Jul 08 18:59:29.610224 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.610200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:29.610323 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.610261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:29.610503 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.610481    2563 scope.go:117] "RemoveContainer" containerID="2b02339573437df431e81bd7cbc50d5c6caeb5b612a59c81f1cbeaf24ce0ba19"
Jul 08 18:59:29.627462 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:29.627442    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:30.449686 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:30.449652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:30.449864 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:30.449770    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:31.415400 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.415357    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlmxj"]
Jul 08 18:59:31.415531 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.415463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:31.415577 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:31.415543    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:31.418474 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.418442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fwx64"]
Jul 08 18:59:31.418646 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.418576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:31.418717 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:31.418690    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:31.419068 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.419049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r6pt5"]
Jul 08 18:59:31.419172 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.419136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:31.419249 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:31.419220    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:31.616132 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.616056    2563 generic.go:356] "Generic (PLEG): container finished" podID="43f058dd-d85e-4c3c-9212-3e7095606619" containerID="294d7bf67833f6f6c9a190f2c5b314582cdb4470fab6d4459b35f6c14958818d" exitCode=0
Jul 08 18:59:31.616132 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.616134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerDied","Data":"294d7bf67833f6f6c9a190f2c5b314582cdb4470fab6d4459b35f6c14958818d"}
Jul 08 18:59:31.620086 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.619922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 18:59:31.620445 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.620420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" event={"ID":"304f1fb3-642c-4933-bc87-3db8e0afbefc","Type":"ContainerStarted","Data":"8da89a41f500f75a16611d40192e4b4002d41f0305982609f5212f74986baecd"}
Jul 08 18:59:31.620769 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.620750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:31.636056 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.636037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 18:59:31.677476 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:31.677427    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8" podStartSLOduration=10.124659526 podStartE2EDuration="29.677414183s" totalImagesPullingTime="19.552754657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:31.677083748 +0000 UTC m=+30.321104478" watchObservedRunningTime="2026-07-08 18:59:31.677414183 +0000 UTC m=+30.321434928"
Jul 08 18:59:32.350099 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.350068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:32.350268 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.350231    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 18:59:32.350701 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.350677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dls7k"
Jul 08 18:59:32.447507 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.447472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:32.567231 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.567206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:32.625341 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.625267    2563 generic.go:356] "Generic (PLEG): container finished" podID="43f058dd-d85e-4c3c-9212-3e7095606619" containerID="1cdebbe94ea9e1c468ef7a2af166e6ca27eb9ccb560d4cb96ece7ae3196720e2" exitCode=0
Jul 08 18:59:32.625790 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.625358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerDied","Data":"1cdebbe94ea9e1c468ef7a2af166e6ca27eb9ccb560d4cb96ece7ae3196720e2"}
Jul 08 18:59:32.631252 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:32.631231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:33.449478 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:33.449447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:33.449597 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:33.449558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:33.449597 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:33.449571    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:33.449597 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:33.449598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:33.449771 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:33.449717    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:33.449852 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:33.449821    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:33.630268 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:33.630237    2563 generic.go:356] "Generic (PLEG): container finished" podID="43f058dd-d85e-4c3c-9212-3e7095606619" containerID="752260808b4ea99d2533f6b09df0f3fee2ff2fe851c65e487682ce8ed042b300" exitCode=0
Jul 08 18:59:33.630703 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:33.630319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerDied","Data":"752260808b4ea99d2533f6b09df0f3fee2ff2fe851c65e487682ce8ed042b300"}
Jul 08 18:59:35.449324 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:35.449284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:35.449892 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:35.449362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:35.449892 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:35.449461    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:35.449892 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:35.449480    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:35.449892 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:35.449538    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:35.449892 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:35.449608    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:35.971751 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:35.971716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:35.971898 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:35.971767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:35.971898 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:35.971886    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:35.972002 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:35.971951    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 19:00:07.971931854 +0000 UTC m=+66.615952567 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 18:59:35.972002 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:35.971887    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:35.972117 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:35.972033    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret podName:e43d8623-63ae-4e5b-92e7-fb1db07f2cec nodeName:}" failed. No retries permitted until 2026-07-08 19:00:07.972016885 +0000 UTC m=+66.616037596 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret") pod "global-pull-secret-syncer-mlmxj" (UID: "e43d8623-63ae-4e5b-92e7-fb1db07f2cec") : object "kube-system"/"original-pull-secret" not registered
Jul 08 18:59:36.072871 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:36.072838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:36.073005 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:36.072974    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 18:59:36.073005 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:36.072990    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 18:59:36.073005 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:36.072999    2563 projected.go:196] Error preparing data for projected volume kube-api-access-mzmld for pod openshift-network-diagnostics/network-check-target-fwx64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:36.073119 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:36.073046    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld podName:252c1061-ec85-4b07-9a88-d0a796e1eac4 nodeName:}" failed. No retries permitted until 2026-07-08 19:00:08.073032875 +0000 UTC m=+66.717053587 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzmld" (UniqueName: "kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld") pod "network-check-target-fwx64" (UID: "252c1061-ec85-4b07-9a88-d0a796e1eac4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 18:59:37.449561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.449487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:37.449561 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.449519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:37.449972 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:37.449593    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 18:59:37.449972 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.449611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:37.449972 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:37.449705    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlmxj" podUID="e43d8623-63ae-4e5b-92e7-fb1db07f2cec"
Jul 08 18:59:37.449972 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:37.449779    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fwx64" podUID="252c1061-ec85-4b07-9a88-d0a796e1eac4"
Jul 08 18:59:37.633667 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.633613    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-89.ec2.internal" event="NodeReady"
Jul 08 18:59:37.633840 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.633780    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 18:59:37.685143 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.685116    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8rz2p"]
Jul 08 18:59:37.689864 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.689849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.691536 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.691513    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dn77p"]
Jul 08 18:59:37.692793 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.692769    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 18:59:37.692892 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.692868    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 18:59:37.692942 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.692908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-97fdz\""
Jul 08 18:59:37.697949 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.697928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:37.702039 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.701411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 18:59:37.702039 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.701337    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 18:59:37.702039 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.701736    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 18:59:37.702039 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.701845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hs7p8\""
Jul 08 18:59:37.703012 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.702966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8rz2p"]
Jul 08 18:59:37.707430 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.707410    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn77p"]
Jul 08 18:59:37.783824 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.783787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.783824 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.783827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21f6bbf-67fd-46e4-aff0-291624037c2c-tmp-dir\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.784033 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.783921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f21f6bbf-67fd-46e4-aff0-291624037c2c-config-volume\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.784033 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.783968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkps\" (UniqueName: \"kubernetes.io/projected/488e72ac-a322-4fb9-8553-936a24cc6683-kube-api-access-8mkps\") pod \"ingress-canary-dn77p\" (UID: \"488e72ac-a322-4fb9-8553-936a24cc6683\") " pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:37.784112 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.784050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/488e72ac-a322-4fb9-8553-936a24cc6683-cert\") pod \"ingress-canary-dn77p\" (UID: \"488e72ac-a322-4fb9-8553-936a24cc6683\") " pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:37.784112 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.784081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhm5\" (UniqueName: \"kubernetes.io/projected/f21f6bbf-67fd-46e4-aff0-291624037c2c-kube-api-access-7zhm5\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.885319 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.885080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.885515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.885336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21f6bbf-67fd-46e4-aff0-291624037c2c-tmp-dir\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.885515 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:37.885255    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 18:59:37.885515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.885403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f21f6bbf-67fd-46e4-aff0-291624037c2c-config-volume\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.885515 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:37.885451    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 18:59:38.385424562 +0000 UTC m=+37.029445280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 18:59:37.885515 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.885491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkps\" (UniqueName: \"kubernetes.io/projected/488e72ac-a322-4fb9-8553-936a24cc6683-kube-api-access-8mkps\") pod \"ingress-canary-dn77p\" (UID: \"488e72ac-a322-4fb9-8553-936a24cc6683\") " pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:37.885833 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.885539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/488e72ac-a322-4fb9-8553-936a24cc6683-cert\") pod \"ingress-canary-dn77p\" (UID: \"488e72ac-a322-4fb9-8553-936a24cc6683\") " pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:37.885833 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.885561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhm5\" (UniqueName: \"kubernetes.io/projected/f21f6bbf-67fd-46e4-aff0-291624037c2c-kube-api-access-7zhm5\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.885833 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.885833    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21f6bbf-67fd-46e4-aff0-291624037c2c-tmp-dir\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.886058 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.886020    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f21f6bbf-67fd-46e4-aff0-291624037c2c-config-volume\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.890868 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.890839    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/488e72ac-a322-4fb9-8553-936a24cc6683-cert\") pod \"ingress-canary-dn77p\" (UID: \"488e72ac-a322-4fb9-8553-936a24cc6683\") " pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:37.895663 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.895502    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhm5\" (UniqueName: \"kubernetes.io/projected/f21f6bbf-67fd-46e4-aff0-291624037c2c-kube-api-access-7zhm5\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:37.895818 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:37.895796    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkps\" (UniqueName: \"kubernetes.io/projected/488e72ac-a322-4fb9-8553-936a24cc6683-kube-api-access-8mkps\") pod \"ingress-canary-dn77p\" (UID: \"488e72ac-a322-4fb9-8553-936a24cc6683\") " pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:38.011330 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:38.011296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn77p"
Jul 08 18:59:38.388944 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:38.388857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:38.389138 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:38.389018    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 18:59:38.389138 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:38.389106    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 18:59:39.389084742 +0000 UTC m=+38.033105455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 18:59:39.337193 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.337162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:39.396373 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.396345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:39.396519 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:39.396458    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 18:59:39.396519 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:39.396508    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 18:59:41.396494381 +0000 UTC m=+40.040515089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 18:59:39.398130 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.398105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:39.450052 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.450030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 18:59:39.450052 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.450041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 18:59:39.450197 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.450032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 18:59:39.453192 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.453166    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 18:59:39.453322 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.453280    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 18:59:39.454802 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.454780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 18:59:39.454902 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.454782    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 18:59:39.454902 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.454874    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7bzch\""
Jul 08 18:59:39.454902 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.454897    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvgdf\""
Jul 08 18:59:39.497940 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.497919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn77p"]
Jul 08 18:59:39.547259 ip-10-0-128-89 kubenswrapper[2563]: W0708 18:59:39.547236    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488e72ac_a322_4fb9_8553_936a24cc6683.slice/crio-3c0db906df59e6ad1fda2ba0a42c10afb9ee3edf47bb8404f7ed9f8f9b54acf7 WatchSource:0}: Error finding container 3c0db906df59e6ad1fda2ba0a42c10afb9ee3edf47bb8404f7ed9f8f9b54acf7: Status 404 returned error can't find the container with id 3c0db906df59e6ad1fda2ba0a42c10afb9ee3edf47bb8404f7ed9f8f9b54acf7
Jul 08 18:59:39.646432 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:39.646399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn77p" event={"ID":"488e72ac-a322-4fb9-8553-936a24cc6683","Type":"ContainerStarted","Data":"3c0db906df59e6ad1fda2ba0a42c10afb9ee3edf47bb8404f7ed9f8f9b54acf7"}
Jul 08 18:59:40.394191 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:40.393946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:40.455172 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:40.455145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:40.655325 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:40.655241    2563 generic.go:356] "Generic (PLEG): container finished" podID="43f058dd-d85e-4c3c-9212-3e7095606619" containerID="05ccc176ce705ab74816452e766b47e8042ad58593de65c758e8abe0b539e58b" exitCode=0
Jul 08 18:59:40.655325 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:40.655309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerDied","Data":"05ccc176ce705ab74816452e766b47e8042ad58593de65c758e8abe0b539e58b"}
Jul 08 18:59:41.410084 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:41.410047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:41.410590 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:41.410257    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 18:59:41.410590 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:41.410350    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 18:59:45.410327797 +0000 UTC m=+44.054348505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 18:59:41.662688 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:41.662596    2563 generic.go:356] "Generic (PLEG): container finished" podID="43f058dd-d85e-4c3c-9212-3e7095606619" containerID="06ba053440cdeb9b316802b20c4eb3f41f7e6747510e8e47d789fedee6154a23" exitCode=0
Jul 08 18:59:41.662688 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:41.662668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerDied","Data":"06ba053440cdeb9b316802b20c4eb3f41f7e6747510e8e47d789fedee6154a23"}
Jul 08 18:59:41.801855 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:41.801824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:41.906004 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:41.905974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:42.609956 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.609924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:42.668214 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.668132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" event={"ID":"43f058dd-d85e-4c3c-9212-3e7095606619","Type":"ContainerStarted","Data":"38f3c5f1c0cfef5e02c9b215909a8581b0eba24daf2c7123ff2d3d9997e9a991"}
Jul 08 18:59:42.669533 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.669509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn77p" event={"ID":"488e72ac-a322-4fb9-8553-936a24cc6683","Type":"ContainerStarted","Data":"c5a1cdc9ee0d2e9e603f3c4188b702738414835a97fb0e3c54c7115839249d8b"}
Jul 08 18:59:42.698259 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.698205    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5m9cq" podStartSLOduration=12.350798151 podStartE2EDuration="40.698188109s" totalImagesPullingTime="28.347389958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:42.695972569 +0000 UTC m=+41.339993300" watchObservedRunningTime="2026-07-08 18:59:42.698188109 +0000 UTC m=+41.342208839"
Jul 08 18:59:42.700777 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.700754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:42.714036 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.713999    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dn77p" podStartSLOduration=2.905921379 podStartE2EDuration="5.713985614s" totalImagesPullingTime="2.808064235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:59:42.713531395 +0000 UTC m=+41.357552126" watchObservedRunningTime="2026-07-08 18:59:42.713985614 +0000 UTC m=+41.358006321"
Jul 08 18:59:42.838684 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.838651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:42.909205 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:42.909176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:45.439788 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:45.439746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:45.440192 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:45.439891    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 18:59:45.440192 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:45.439951    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 18:59:53.439937322 +0000 UTC m=+52.083958030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 18:59:53.494533 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:53.494495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 18:59:53.494994 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:53.494664    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 18:59:53.494994 ip-10-0-128-89 kubenswrapper[2563]: E0708 18:59:53.494744    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 19:00:09.494728287 +0000 UTC m=+68.138748995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 18:59:54.558424 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:54.558390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 18:59:54.657486 ip-10-0-128-89 kubenswrapper[2563]: I0708 18:59:54.657452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:02.817575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.817540    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"]
Jul 08 19:00:02.848714 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.848682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"]
Jul 08 19:00:02.848861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.848806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.851902 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.851879    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 08 19:00:02.852228 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.852210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 19:00:02.852453 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.852437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 19:00:02.853526 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.853508    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 08 19:00:02.853638 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.853549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 08 19:00:02.854027 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 19:00:02.854766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-hub\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.854766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 08 19:00:02.854766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.854766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.854766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-ca\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.854766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753956b-81f7-4f49-b651-0424b0191f2c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.854766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.854676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8v4\" (UniqueName: \"kubernetes.io/projected/4753956b-81f7-4f49-b651-0424b0191f2c-kube-api-access-nr8v4\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.955919 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.955878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-ca\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.956106 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.955930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753956b-81f7-4f49-b651-0424b0191f2c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.956106 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.955947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8v4\" (UniqueName: \"kubernetes.io/projected/4753956b-81f7-4f49-b651-0424b0191f2c-kube-api-access-nr8v4\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.956106 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.955987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-hub\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.956106 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.956016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.956106 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.956033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.956724 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.956690    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4753956b-81f7-4f49-b651-0424b0191f2c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.959513 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.959488    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-ca\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.959617 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.959526    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-hub\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.959617 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.959579    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.959738 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.959664    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4753956b-81f7-4f49-b651-0424b0191f2c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:02.966999 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:02.966976    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8v4\" (UniqueName: \"kubernetes.io/projected/4753956b-81f7-4f49-b651-0424b0191f2c-kube-api-access-nr8v4\") pod \"cluster-proxy-proxy-agent-85c878844-z8blf\" (UID: \"4753956b-81f7-4f49-b651-0424b0191f2c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:03.167892 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.167796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:00:03.292932 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.292898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"]
Jul 08 19:00:03.295705 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:00:03.295675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4753956b_81f7_4f49_b651_0424b0191f2c.slice/crio-bb6c7f7122cd2720a1ab9f94a8e9eac6878ef4d86425ba6cf8655ac28674a624 WatchSource:0}: Error finding container bb6c7f7122cd2720a1ab9f94a8e9eac6878ef4d86425ba6cf8655ac28674a624: Status 404 returned error can't find the container with id bb6c7f7122cd2720a1ab9f94a8e9eac6878ef4d86425ba6cf8655ac28674a624
Jul 08 19:00:03.637834 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.637802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:03.640226 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.640207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-55wp8"
Jul 08 19:00:03.715145 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.715109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" event={"ID":"4753956b-81f7-4f49-b651-0424b0191f2c","Type":"ContainerStarted","Data":"bb6c7f7122cd2720a1ab9f94a8e9eac6878ef4d86425ba6cf8655ac28674a624"}
Jul 08 19:00:03.739803 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.739776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:03.858527 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.858500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:03.968864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:03.968793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:07.994006 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:07.993948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:00:07.994006 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:07.994003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 19:00:07.997244 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:07.997217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 19:00:07.997373 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:07.997217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 19:00:08.004735 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:00:08.004719    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 19:00:08.004785 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:00:08.004769    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 19:01:12.004754375 +0000 UTC m=+130.648775083 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : secret "metrics-daemon-secret" not found
Jul 08 19:00:08.006884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.006864    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e43d8623-63ae-4e5b-92e7-fb1db07f2cec-original-pull-secret\") pod \"global-pull-secret-syncer-mlmxj\" (UID: \"e43d8623-63ae-4e5b-92e7-fb1db07f2cec\") " pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 19:00:08.094412 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.094359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 19:00:08.097981 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.097965    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 19:00:08.107464 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.107442    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 19:00:08.117998 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.117977    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmld\" (UniqueName: \"kubernetes.io/projected/252c1061-ec85-4b07-9a88-d0a796e1eac4-kube-api-access-mzmld\") pod \"network-check-target-fwx64\" (UID: \"252c1061-ec85-4b07-9a88-d0a796e1eac4\") " pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 19:00:08.264666 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.264560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7bzch\""
Jul 08 19:00:08.273033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.273013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 19:00:08.273033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.273027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlmxj"
Jul 08 19:00:08.401607 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.401579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fwx64"]
Jul 08 19:00:08.406162 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:00:08.406135    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252c1061_ec85_4b07_9a88_d0a796e1eac4.slice/crio-75a0d2d73dac7061a9d906e4260b3da8dbbde52a4ca6f0ea64e38c09d13febc3 WatchSource:0}: Error finding container 75a0d2d73dac7061a9d906e4260b3da8dbbde52a4ca6f0ea64e38c09d13febc3: Status 404 returned error can't find the container with id 75a0d2d73dac7061a9d906e4260b3da8dbbde52a4ca6f0ea64e38c09d13febc3
Jul 08 19:00:08.416732 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.416706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlmxj"]
Jul 08 19:00:08.419611 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:00:08.419585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43d8623_63ae_4e5b_92e7_fb1db07f2cec.slice/crio-4fc22233cf8491eec60f8a69aed5a10e70ba5cf20b8cd726deb73650b520683b WatchSource:0}: Error finding container 4fc22233cf8491eec60f8a69aed5a10e70ba5cf20b8cd726deb73650b520683b: Status 404 returned error can't find the container with id 4fc22233cf8491eec60f8a69aed5a10e70ba5cf20b8cd726deb73650b520683b
Jul 08 19:00:08.733673 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.733556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlmxj" event={"ID":"e43d8623-63ae-4e5b-92e7-fb1db07f2cec","Type":"ContainerStarted","Data":"4fc22233cf8491eec60f8a69aed5a10e70ba5cf20b8cd726deb73650b520683b"}
Jul 08 19:00:08.734916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:08.734885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fwx64" event={"ID":"252c1061-ec85-4b07-9a88-d0a796e1eac4","Type":"ContainerStarted","Data":"75a0d2d73dac7061a9d906e4260b3da8dbbde52a4ca6f0ea64e38c09d13febc3"}
Jul 08 19:00:09.200918 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:09.200881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:09.505360 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:09.505184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:00:09.505553 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:00:09.505384    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 19:00:09.505553 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:00:09.505455    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 19:00:41.505433984 +0000 UTC m=+100.149454692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 19:00:14.756397 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.756364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" event={"ID":"4753956b-81f7-4f49-b651-0424b0191f2c","Type":"ContainerStarted","Data":"30c8fba654ae46d3fbf87cedd558816b25c9381496a90f33af3d8a90404c0397"}
Jul 08 19:00:14.756881 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.756408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" event={"ID":"4753956b-81f7-4f49-b651-0424b0191f2c","Type":"ContainerStarted","Data":"2e2a235ac81a730f1427b9a7229b59e631097bb1a8571d9534329c2b3e1006f1"}
Jul 08 19:00:14.756881 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.756428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" event={"ID":"4753956b-81f7-4f49-b651-0424b0191f2c","Type":"ContainerStarted","Data":"4a03a8cdfa8f97abd844873c5ddb48c8b6cf81824f3086abdf36de1be8b3f3e9"}
Jul 08 19:00:14.757655 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.757614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlmxj" event={"ID":"e43d8623-63ae-4e5b-92e7-fb1db07f2cec","Type":"ContainerStarted","Data":"c2e061b78f2253d5b9339d52be43ce31c7c70eab90b2909d12c3b874ef6a638f"}
Jul 08 19:00:14.759062 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.759041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fwx64" event={"ID":"252c1061-ec85-4b07-9a88-d0a796e1eac4","Type":"ContainerStarted","Data":"274ebd5d9211ec73098456b509ad349fcf4e97ce63a4309c5e0c4382a814835a"}
Jul 08 19:00:14.759145 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.759087    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 19:00:14.780762 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.780710    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" podStartSLOduration=1.859703384 podStartE2EDuration="12.780695923s" totalImagesPullingTime="10.920992539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:00:14.780012976 +0000 UTC m=+73.424033705" watchObservedRunningTime="2026-07-08 19:00:14.780695923 +0000 UTC m=+73.424716631"
Jul 08 19:00:14.796787 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.796741    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mlmxj" podStartSLOduration=66.988875286 podStartE2EDuration="1m12.796726217s" totalImagesPullingTime="5.807850931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:00:14.795859812 +0000 UTC m=+73.439880543" watchObservedRunningTime="2026-07-08 19:00:14.796726217 +0000 UTC m=+73.440746946"
Jul 08 19:00:14.813022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:14.812980    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fwx64" podStartSLOduration=66.99221938 podStartE2EDuration="1m12.812965818s" totalImagesPullingTime="5.820746438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:00:14.812081197 +0000 UTC m=+73.456101927" watchObservedRunningTime="2026-07-08 19:00:14.812965818 +0000 UTC m=+73.456986548"
Jul 08 19:00:32.888135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:32.888088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:32.952876 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:32.952846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:41.532740 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:41.532698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:00:41.533135 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:00:41.532841    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 19:00:41.533135 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:00:41.532894    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 19:01:45.532880467 +0000 UTC m=+164.176901175 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 19:00:44.132548 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:44.132514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:00:45.768542 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:00:45.768510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fwx64"
Jul 08 19:01:12.025902 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:01:12.025845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:01:12.026387 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:01:12.025995    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 19:01:12.026387 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:01:12.026069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 19:03:14.026051753 +0000 UTC m=+252.670072461 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : secret "metrics-daemon-secret" not found
Jul 08 19:01:40.699362 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:01:40.699312    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8rz2p" podUID="f21f6bbf-67fd-46e4-aff0-291624037c2c"
Jul 08 19:01:41.002976 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:01:41.002941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:01:42.468510 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:01:42.468474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 19:01:45.555716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:01:45.555681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:01:45.556084 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:01:45.555835    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 19:01:45.556084 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:01:45.555908    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls podName:f21f6bbf-67fd-46e4-aff0-291624037c2c nodeName:}" failed. No retries permitted until 2026-07-08 19:03:47.555891678 +0000 UTC m=+286.199912389 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls") pod "dns-default-8rz2p" (UID: "f21f6bbf-67fd-46e4-aff0-291624037c2c") : secret "dns-default-metrics-tls" not found
Jul 08 19:01:54.450029 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:01:54.449948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:03:14.053777 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:14.053728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:03:14.054278 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:14.053854    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 19:03:14.054278 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:14.053908    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs podName:daca3b10-16da-4559-8498-a0bfec82aad8 nodeName:}" failed. No retries permitted until 2026-07-08 19:05:16.053892801 +0000 UTC m=+374.697913509 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs") pod "network-metrics-daemon-r6pt5" (UID: "daca3b10-16da-4559-8498-a0bfec82aad8") : secret "metrics-daemon-secret" not found
Jul 08 19:03:17.813831 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:17.813799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:17.898457 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:17.898426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:19.812643 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.812607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-m929j"]
Jul 08 19:03:19.816653 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.815210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.820154 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.820118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bwhf9\""
Jul 08 19:03:19.820521 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.820498    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 08 19:03:19.821583 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.821564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 08 19:03:19.821709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.821565    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 08 19:03:19.821709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.821593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 08 19:03:19.829199 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.829179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-m929j"]
Jul 08 19:03:19.841234 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.841210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 08 19:03:19.892144 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.892118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cd19b-8af5-479c-8bbd-47402f98e466-trusted-ca\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.892144 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.892146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cd19b-8af5-479c-8bbd-47402f98e466-serving-cert\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.892304 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.892187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1cd19b-8af5-479c-8bbd-47402f98e466-config\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.892304 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.892239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t49cj\" (UniqueName: \"kubernetes.io/projected/8d1cd19b-8af5-479c-8bbd-47402f98e466-kube-api-access-t49cj\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.992545 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.992505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cd19b-8af5-479c-8bbd-47402f98e466-trusted-ca\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.992545 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.992544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cd19b-8af5-479c-8bbd-47402f98e466-serving-cert\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.992832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.992584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1cd19b-8af5-479c-8bbd-47402f98e466-config\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.992832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.992615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t49cj\" (UniqueName: \"kubernetes.io/projected/8d1cd19b-8af5-479c-8bbd-47402f98e466-kube-api-access-t49cj\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.993299 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.993277    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cd19b-8af5-479c-8bbd-47402f98e466-trusted-ca\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.993380 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.993277    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d1cd19b-8af5-479c-8bbd-47402f98e466-config\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:19.994989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:19.994967    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cd19b-8af5-479c-8bbd-47402f98e466-serving-cert\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:20.002218 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:20.002198    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t49cj\" (UniqueName: \"kubernetes.io/projected/8d1cd19b-8af5-479c-8bbd-47402f98e466-kube-api-access-t49cj\") pod \"console-operator-68b757fb67-m929j\" (UID: \"8d1cd19b-8af5-479c-8bbd-47402f98e466\") " pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:20.040295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:20.040271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:20.124827 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:20.124742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:20.151127 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:20.151091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:20.248375 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:20.248352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-m929j"]
Jul 08 19:03:20.250590 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:03:20.250560    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1cd19b_8af5_479c_8bbd_47402f98e466.slice/crio-f57171a8df0348b352775b9985ca5efbe422903800900646f4b04302e9262810 WatchSource:0}: Error finding container f57171a8df0348b352775b9985ca5efbe422903800900646f4b04302e9262810: Status 404 returned error can't find the container with id f57171a8df0348b352775b9985ca5efbe422903800900646f4b04302e9262810
Jul 08 19:03:20.268104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:20.268081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-m929j" event={"ID":"8d1cd19b-8af5-479c-8bbd-47402f98e466","Type":"ContainerStarted","Data":"f57171a8df0348b352775b9985ca5efbe422903800900646f4b04302e9262810"}
Jul 08 19:03:22.119799 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:22.119767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:22.183357 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:22.183320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:23.280929 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:23.280864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/0.log"
Jul 08 19:03:23.280929 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:23.280907    2563 generic.go:356] "Generic (PLEG): container finished" podID="8d1cd19b-8af5-479c-8bbd-47402f98e466" containerID="18d88a8a00a10eb5aa6365fff9d9f4cf7baa97ab78ee7181d5f095d80dc5568e" exitCode=255
Jul 08 19:03:23.281292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:23.280953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-m929j" event={"ID":"8d1cd19b-8af5-479c-8bbd-47402f98e466","Type":"ContainerDied","Data":"18d88a8a00a10eb5aa6365fff9d9f4cf7baa97ab78ee7181d5f095d80dc5568e"}
Jul 08 19:03:23.281292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:23.281214    2563 scope.go:117] "RemoveContainer" containerID="18d88a8a00a10eb5aa6365fff9d9f4cf7baa97ab78ee7181d5f095d80dc5568e"
Jul 08 19:03:24.215362 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.215330    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wrzk6"]
Jul 08 19:03:24.218534 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.218502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.222213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.222187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 19:03:24.222522 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.222503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 19:03:24.223555 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.223535    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 19:03:24.223684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.223577    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v5zgs\""
Jul 08 19:03:24.223684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.223593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 19:03:24.257209 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.257184    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wrzk6"]
Jul 08 19:03:24.285280 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.285255    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/1.log"
Jul 08 19:03:24.285692 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.285614    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/0.log"
Jul 08 19:03:24.285692 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.285669    2563 generic.go:356] "Generic (PLEG): container finished" podID="8d1cd19b-8af5-479c-8bbd-47402f98e466" containerID="746829f542c8cc048774b6e06d99ea3fa4c37ed6057e2b0962189e11faede4e1" exitCode=255
Jul 08 19:03:24.285783 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.285746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-m929j" event={"ID":"8d1cd19b-8af5-479c-8bbd-47402f98e466","Type":"ContainerDied","Data":"746829f542c8cc048774b6e06d99ea3fa4c37ed6057e2b0962189e11faede4e1"}
Jul 08 19:03:24.285823 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.285811    2563 scope.go:117] "RemoveContainer" containerID="18d88a8a00a10eb5aa6365fff9d9f4cf7baa97ab78ee7181d5f095d80dc5568e"
Jul 08 19:03:24.286000 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.285985    2563 scope.go:117] "RemoveContainer" containerID="746829f542c8cc048774b6e06d99ea3fa4c37ed6057e2b0962189e11faede4e1"
Jul 08 19:03:24.286187 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.286162    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-m929j_openshift-console-operator(8d1cd19b-8af5-479c-8bbd-47402f98e466)\"" pod="openshift-console-operator/console-operator-68b757fb67-m929j" podUID="8d1cd19b-8af5-479c-8bbd-47402f98e466"
Jul 08 19:03:24.325432 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.325414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70e1619c-bd2b-40ef-a1d5-91954ea233dd-data-volume\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.325552 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.325461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd794\" (UniqueName: \"kubernetes.io/projected/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-api-access-pd794\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.325552 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.325488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70e1619c-bd2b-40ef-a1d5-91954ea233dd-crio-socket\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.325702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.325573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.325702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.325656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.357233 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.357207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:24.426437 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.426405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd794\" (UniqueName: \"kubernetes.io/projected/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-api-access-pd794\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.426437 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.426439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70e1619c-bd2b-40ef-a1d5-91954ea233dd-crio-socket\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.426462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.426544    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.426555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.426617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70e1619c-bd2b-40ef-a1d5-91954ea233dd-crio-socket\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.426644    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:24.926602381 +0000 UTC m=+263.570623089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : configmap "kube-rbac-proxy" not found
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.426694    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.426720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70e1619c-bd2b-40ef-a1d5-91954ea233dd-data-volume\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.426884 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.426753    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:24.92673643 +0000 UTC m=+263.570757142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:24.427145 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.427097    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70e1619c-bd2b-40ef-a1d5-91954ea233dd-data-volume\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.429756 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.429737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:24.436086 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.436065    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd794\" (UniqueName: \"kubernetes.io/projected/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-api-access-pd794\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.931882 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.931850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.932079 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:24.931895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:24.932079 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.931983    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 19:03:24.932079 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.931998    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:24.932079 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.932057    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:25.932043802 +0000 UTC m=+264.576064511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:24.932079 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:24.932069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:25.932063164 +0000 UTC m=+264.576083872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : configmap "kube-rbac-proxy" not found
Jul 08 19:03:25.289930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.289909    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/1.log"
Jul 08 19:03:25.290270 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.290218    2563 scope.go:117] "RemoveContainer" containerID="746829f542c8cc048774b6e06d99ea3fa4c37ed6057e2b0962189e11faede4e1"
Jul 08 19:03:25.290367 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:25.290348    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-m929j_openshift-console-operator(8d1cd19b-8af5-479c-8bbd-47402f98e466)\"" pod="openshift-console-operator/console-operator-68b757fb67-m929j" podUID="8d1cd19b-8af5-479c-8bbd-47402f98e466"
Jul 08 19:03:25.544968 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.544893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:25.631601 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.631567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:25.731809 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.731777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:25.940185 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.940077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:25.940185 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:25.940153    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 19:03:25.940402 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:25.940212    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:27.940195945 +0000 UTC m=+266.584216653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : configmap "kube-rbac-proxy" not found
Jul 08 19:03:25.940402 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.940149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:25.940402 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:25.940238    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:25.940402 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:25.940309    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:27.940292346 +0000 UTC m=+266.584313057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:25.989525 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:25.989501    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n4x4s_4fea7aea-3311-4b3a-bbf9-6a0c2585aec6/dns-node-resolver/0.log"
Jul 08 19:03:26.658292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:26.658255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:26.759056 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:26.759031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:26.989482 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:26.989461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2ztqx_77bd5f3c-b7d3-4a92-b408-8366970336ef/node-ca/0.log"
Jul 08 19:03:27.095376 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:27.095342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:27.357700 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:27.357614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:27.459900 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:27.459872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:27.789572 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:27.789547    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn77p_488e72ac-a322-4fb9-8553-936a24cc6683/serve-healthcheck-canary/0.log"
Jul 08 19:03:27.955412 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:27.955383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:27.955541 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:27.955426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:27.955541 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:27.955504    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 19:03:27.955541 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:27.955526    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:27.955695 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:27.955565    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:31.955549242 +0000 UTC m=+270.599569950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : configmap "kube-rbac-proxy" not found
Jul 08 19:03:27.955695 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:27.955579    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:31.955572973 +0000 UTC m=+270.599593681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:28.110249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.110165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:28.165941 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.165908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:28.960830 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.960794    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-gnqgl"]
Jul 08 19:03:28.964941 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.964914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:28.968076 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.968056    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 08 19:03:28.968195 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.968099    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 08 19:03:28.969350 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.969333    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 08 19:03:28.969431 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.969368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mgm4t\""
Jul 08 19:03:28.969431 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.969374    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 08 19:03:28.972791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:28.972771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-gnqgl"]
Jul 08 19:03:29.000463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.000434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:29.060439 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.060410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:29.065501 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.065479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c246a81-ad90-4ed7-b1be-df868990fb07-signing-key\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.065651 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.065566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c246a81-ad90-4ed7-b1be-df868990fb07-signing-cabundle\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.065651 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.065645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tv5w\" (UniqueName: \"kubernetes.io/projected/2c246a81-ad90-4ed7-b1be-df868990fb07-kube-api-access-6tv5w\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.166759 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.166724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tv5w\" (UniqueName: \"kubernetes.io/projected/2c246a81-ad90-4ed7-b1be-df868990fb07-kube-api-access-6tv5w\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.166916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.166768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c246a81-ad90-4ed7-b1be-df868990fb07-signing-key\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.166916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.166852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c246a81-ad90-4ed7-b1be-df868990fb07-signing-cabundle\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.167518 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.167496    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c246a81-ad90-4ed7-b1be-df868990fb07-signing-cabundle\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.169312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.169282    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c246a81-ad90-4ed7-b1be-df868990fb07-signing-key\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.177066 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.177041    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tv5w\" (UniqueName: \"kubernetes.io/projected/2c246a81-ad90-4ed7-b1be-df868990fb07-kube-api-access-6tv5w\") pod \"service-ca-758f599c5-gnqgl\" (UID: \"2c246a81-ad90-4ed7-b1be-df868990fb07\") " pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.273927 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.273901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-gnqgl"
Jul 08 19:03:29.393901 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:29.393861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-gnqgl"]
Jul 08 19:03:29.397200 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:03:29.397174    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c246a81_ad90_4ed7_b1be_df868990fb07.slice/crio-9df6cd223cc6db3706d2b9ca6df8036fed87142681583515f0dca979188a1a05 WatchSource:0}: Error finding container 9df6cd223cc6db3706d2b9ca6df8036fed87142681583515f0dca979188a1a05: Status 404 returned error can't find the container with id 9df6cd223cc6db3706d2b9ca6df8036fed87142681583515f0dca979188a1a05
Jul 08 19:03:30.055665 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:30.055616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:30.124990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:30.124890    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:30.124990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:30.124961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:30.125416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:30.125397    2563 scope.go:117] "RemoveContainer" containerID="746829f542c8cc048774b6e06d99ea3fa4c37ed6057e2b0962189e11faede4e1"
Jul 08 19:03:30.125618 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:30.125593    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-m929j_openshift-console-operator(8d1cd19b-8af5-479c-8bbd-47402f98e466)\"" pod="openshift-console-operator/console-operator-68b757fb67-m929j" podUID="8d1cd19b-8af5-479c-8bbd-47402f98e466"
Jul 08 19:03:30.304713 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:30.304675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-gnqgl" event={"ID":"2c246a81-ad90-4ed7-b1be-df868990fb07","Type":"ContainerStarted","Data":"9df6cd223cc6db3706d2b9ca6df8036fed87142681583515f0dca979188a1a05"}
Jul 08 19:03:31.987396 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:31.987360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:31.987873 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:31.987405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:31.987873 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:31.987492    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 19:03:31.987873 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:31.987521    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:31.987873 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:31.987566    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:39.987546395 +0000 UTC m=+278.631567112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : configmap "kube-rbac-proxy" not found
Jul 08 19:03:31.987873 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:31.987588    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:39.987577464 +0000 UTC m=+278.631598177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:33.314304 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:33.314265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-gnqgl" event={"ID":"2c246a81-ad90-4ed7-b1be-df868990fb07","Type":"ContainerStarted","Data":"0a1344c3fb0eb077b3dc2f8a60969c98ab703eaffc31c0e435e678325ff19f7d"}
Jul 08 19:03:33.399805 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:33.399771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:34.083611 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:34.083554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:34.286174 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:34.286142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:34.383980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:34.383893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:34.484375 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:34.484341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:34.753845 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:34.753813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:35.990923 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:35.990892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:36.137920 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:36.137891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:36.328674 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:36.328565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:36.540837 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:36.540793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:36.737438 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:36.737343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:37.482939 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:37.482908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:37.585231 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:37.585199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:37.953053 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:37.953020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:38.023414 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:38.023382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:38.126017 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:38.125987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:38.332954 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:38.332926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:38.542182 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:38.542128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:38.733561 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:38.733479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:39.779048 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:39.779010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:40.051160 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:40.051077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:40.051160 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:40.051123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:40.051334 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:40.051278    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:40.051369 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:40.051347    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls podName:70e1619c-bd2b-40ef-a1d5-91954ea233dd nodeName:}" failed. No retries permitted until 2026-07-08 19:03:56.051328193 +0000 UTC m=+294.695348904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrzk6" (UID: "70e1619c-bd2b-40ef-a1d5-91954ea233dd") : secret "insights-runtime-extractor-tls" not found
Jul 08 19:03:40.051790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:40.051763    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70e1619c-bd2b-40ef-a1d5-91954ea233dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:40.926879 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:40.926850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:41.027173 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:41.027107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:41.734119 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:41.734085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:41.935720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:41.935690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:42.327539 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:42.327509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:42.450898 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:42.450863    2563 scope.go:117] "RemoveContainer" containerID="746829f542c8cc048774b6e06d99ea3fa4c37ed6057e2b0962189e11faede4e1"
Jul 08 19:03:42.532069 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:42.532043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:42.933181 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:42.933146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:43.346460 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.346433    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:03:43.346864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.346822    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/1.log"
Jul 08 19:03:43.346864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.346850    2563 generic.go:356] "Generic (PLEG): container finished" podID="8d1cd19b-8af5-479c-8bbd-47402f98e466" containerID="56e732c5592ccca8d22ca5f12ed04370691cf7a9beb0d91a4b805c7a843612e0" exitCode=255
Jul 08 19:03:43.346948 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.346912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-m929j" event={"ID":"8d1cd19b-8af5-479c-8bbd-47402f98e466","Type":"ContainerDied","Data":"56e732c5592ccca8d22ca5f12ed04370691cf7a9beb0d91a4b805c7a843612e0"}
Jul 08 19:03:43.346989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.346957    2563 scope.go:117] "RemoveContainer" containerID="746829f542c8cc048774b6e06d99ea3fa4c37ed6057e2b0962189e11faede4e1"
Jul 08 19:03:43.347283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.347263    2563 scope.go:117] "RemoveContainer" containerID="56e732c5592ccca8d22ca5f12ed04370691cf7a9beb0d91a4b805c7a843612e0"
Jul 08 19:03:43.347459 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:43.347430    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-m929j_openshift-console-operator(8d1cd19b-8af5-479c-8bbd-47402f98e466)\"" pod="openshift-console-operator/console-operator-68b757fb67-m929j" podUID="8d1cd19b-8af5-479c-8bbd-47402f98e466"
Jul 08 19:03:43.367422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.367380    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-gnqgl" podStartSLOduration=12.387849121 podStartE2EDuration="15.367365187s" totalImagesPullingTime="2.979516066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:33.34073481 +0000 UTC m=+271.984755551" watchObservedRunningTime="2026-07-08 19:03:43.367365187 +0000 UTC m=+282.011385974"
Jul 08 19:03:43.527111 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:43.527081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:44.003853 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:44.003818    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8rz2p" podUID="f21f6bbf-67fd-46e4-aff0-291624037c2c"
Jul 08 19:03:44.148071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:44.148032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:44.351854 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:44.351784    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:03:44.352220 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:44.351867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:03:44.415046 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:44.415014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:44.560453 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:44.560414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:44.756115 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:44.756085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:47.446346 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:47.446314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:47.604515 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:47.604479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:03:47.606808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:47.606781    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f21f6bbf-67fd-46e4-aff0-291624037c2c-metrics-tls\") pod \"dns-default-8rz2p\" (UID: \"f21f6bbf-67fd-46e4-aff0-291624037c2c\") " pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:03:47.656578 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:47.656552    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-97fdz\""
Jul 08 19:03:47.663456 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:47.663441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:03:47.803539 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:47.803515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8rz2p"]
Jul 08 19:03:47.805743 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:03:47.805715    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21f6bbf_67fd_46e4_aff0_291624037c2c.slice/crio-930360e77b1d99d4005b5efa6c2d419f2cd3f61493cd24c6d61e38b0e556764b WatchSource:0}: Error finding container 930360e77b1d99d4005b5efa6c2d419f2cd3f61493cd24c6d61e38b0e556764b: Status 404 returned error can't find the container with id 930360e77b1d99d4005b5efa6c2d419f2cd3f61493cd24c6d61e38b0e556764b
Jul 08 19:03:48.365903 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:48.365865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8rz2p" event={"ID":"f21f6bbf-67fd-46e4-aff0-291624037c2c","Type":"ContainerStarted","Data":"930360e77b1d99d4005b5efa6c2d419f2cd3f61493cd24c6d61e38b0e556764b"}
Jul 08 19:03:50.125457 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:50.125413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:50.125994 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:50.125494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:03:50.125994 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:50.125859    2563 scope.go:117] "RemoveContainer" containerID="56e732c5592ccca8d22ca5f12ed04370691cf7a9beb0d91a4b805c7a843612e0"
Jul 08 19:03:50.126122 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:50.126065    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-m929j_openshift-console-operator(8d1cd19b-8af5-479c-8bbd-47402f98e466)\"" pod="openshift-console-operator/console-operator-68b757fb67-m929j" podUID="8d1cd19b-8af5-479c-8bbd-47402f98e466"
Jul 08 19:03:50.373272 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:50.373181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8rz2p" event={"ID":"f21f6bbf-67fd-46e4-aff0-291624037c2c","Type":"ContainerStarted","Data":"3ad3cfbf66a36e2984b137fa2e57ac6dc0bee1397c8fb086e01946ffed1ac0e2"}
Jul 08 19:03:50.373272 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:50.373227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8rz2p" event={"ID":"f21f6bbf-67fd-46e4-aff0-291624037c2c","Type":"ContainerStarted","Data":"fabf0be74cebffe11bd90d787ff59b16835fecc59fbb7981d9f8cad0b37f4f4f"}
Jul 08 19:03:50.373497 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:50.373315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:03:50.399981 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:50.399935    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8rz2p" podStartSLOduration=251.11636348 podStartE2EDuration="4m13.399920821s" totalImagesPullingTime="2.283557341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:50.399135048 +0000 UTC m=+289.043155779" watchObservedRunningTime="2026-07-08 19:03:50.399920821 +0000 UTC m=+289.043941552"
Jul 08 19:03:51.921191 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.921157    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fd4f585c9-5ftj8"]
Jul 08 19:03:51.923918 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.923903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:51.927031 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.927008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 19:03:51.927867 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.927850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 19:03:51.928010 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.927993    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2n9mw\""
Jul 08 19:03:51.928563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.928549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 19:03:51.937181 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.937159    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 19:03:51.950720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.950695    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd4f585c9-5ftj8"]
Jul 08 19:03:51.986887 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:51.986861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:52.037117 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297d68f5-6075-415b-b5bb-76a120365d16-trusted-ca\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.037117 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/297d68f5-6075-415b-b5bb-76a120365d16-installation-pull-secrets\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.037307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/297d68f5-6075-415b-b5bb-76a120365d16-image-registry-private-configuration\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.037307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/297d68f5-6075-415b-b5bb-76a120365d16-registry-certificates\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.037307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/297d68f5-6075-415b-b5bb-76a120365d16-ca-trust-extracted\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.037307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-registry-tls\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.037307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-bound-sa-token\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.037490 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.037315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4jc\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-kube-api-access-rt4jc\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.101965 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.101935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:52.138492 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/297d68f5-6075-415b-b5bb-76a120365d16-ca-trust-extracted\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.138492 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-registry-tls\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.138739 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-bound-sa-token\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.138739 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4jc\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-kube-api-access-rt4jc\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.138739 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297d68f5-6075-415b-b5bb-76a120365d16-trusted-ca\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.138739 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/297d68f5-6075-415b-b5bb-76a120365d16-installation-pull-secrets\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.138944 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/297d68f5-6075-415b-b5bb-76a120365d16-image-registry-private-configuration\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.138944 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/297d68f5-6075-415b-b5bb-76a120365d16-registry-certificates\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.139041 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.138955    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/297d68f5-6075-415b-b5bb-76a120365d16-ca-trust-extracted\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.139575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.139552    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/297d68f5-6075-415b-b5bb-76a120365d16-registry-certificates\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.139792 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.139768    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/297d68f5-6075-415b-b5bb-76a120365d16-trusted-ca\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.141026 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.141001    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-registry-tls\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.141026 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.141009    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/297d68f5-6075-415b-b5bb-76a120365d16-installation-pull-secrets\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.141170 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.141143    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/297d68f5-6075-415b-b5bb-76a120365d16-image-registry-private-configuration\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.151099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.151073    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4jc\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-kube-api-access-rt4jc\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.151366 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.151348    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/297d68f5-6075-415b-b5bb-76a120365d16-bound-sa-token\") pod \"image-registry-6fd4f585c9-5ftj8\" (UID: \"297d68f5-6075-415b-b5bb-76a120365d16\") " pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.233552 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.233467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:52.364937 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.364911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd4f585c9-5ftj8"]
Jul 08 19:03:52.367156 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:03:52.367128    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297d68f5_6075_415b_b5bb_76a120365d16.slice/crio-be6b9b02046fef7b0f9ab49fe3a71067d51216af78735691d60a4d9abb0dc0f4 WatchSource:0}: Error finding container be6b9b02046fef7b0f9ab49fe3a71067d51216af78735691d60a4d9abb0dc0f4: Status 404 returned error can't find the container with id be6b9b02046fef7b0f9ab49fe3a71067d51216af78735691d60a4d9abb0dc0f4
Jul 08 19:03:52.381364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:52.381342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8" event={"ID":"297d68f5-6075-415b-b5bb-76a120365d16","Type":"ContainerStarted","Data":"be6b9b02046fef7b0f9ab49fe3a71067d51216af78735691d60a4d9abb0dc0f4"}
Jul 08 19:03:53.385333 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:53.385296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8" event={"ID":"297d68f5-6075-415b-b5bb-76a120365d16","Type":"ContainerStarted","Data":"824501935ebac94e92b6475c540257718bf27e1131c25890b26c7beb5a9e862b"}
Jul 08 19:03:53.385776 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:53.385422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:03:53.407799 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:53.407749    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8" podStartSLOduration=2.407734564 podStartE2EDuration="2.407734564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:03:53.407009538 +0000 UTC m=+292.051030269" watchObservedRunningTime="2026-07-08 19:03:53.407734564 +0000 UTC m=+292.051755293"
Jul 08 19:03:54.550717 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:54.550679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:54.650166 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:54.650121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:54.728864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:54.728833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:55.007149 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:55.007117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:56.067887 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:56.067829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:56.070141 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:56.070116    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70e1619c-bd2b-40ef-a1d5-91954ea233dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrzk6\" (UID: \"70e1619c-bd2b-40ef-a1d5-91954ea233dd\") " pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:56.328698 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:56.328582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wrzk6"
Jul 08 19:03:56.456274 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:56.456241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wrzk6"]
Jul 08 19:03:56.460298 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:03:56.460272    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e1619c_bd2b_40ef_a1d5_91954ea233dd.slice/crio-75d035adcd8f0aa68671bf25ed97085872851a2fa05f59a9496bded4adf29097 WatchSource:0}: Error finding container 75d035adcd8f0aa68671bf25ed97085872851a2fa05f59a9496bded4adf29097: Status 404 returned error can't find the container with id 75d035adcd8f0aa68671bf25ed97085872851a2fa05f59a9496bded4adf29097
Jul 08 19:03:57.403182 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:57.403137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrzk6" event={"ID":"70e1619c-bd2b-40ef-a1d5-91954ea233dd","Type":"ContainerStarted","Data":"75d035adcd8f0aa68671bf25ed97085872851a2fa05f59a9496bded4adf29097"}
Jul 08 19:03:57.450708 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:03:57.450672    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-r6pt5" podUID="daca3b10-16da-4559-8498-a0bfec82aad8"
Jul 08 19:03:57.733906 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:57.733801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:57.822806 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:57.822764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:03:58.323056 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:03:58.323019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:00.379744 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:00.379721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8rz2p"
Jul 08 19:04:00.415884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:00.415852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrzk6" event={"ID":"70e1619c-bd2b-40ef-a1d5-91954ea233dd","Type":"ContainerStarted","Data":"1478f3f2f2ab2093fe21b4743f8dfcbf7f8f9b000228361a261ace7cff0717ff"}
Jul 08 19:04:01.284369 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:01.284331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:01.341791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:01.341752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:01.712717 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:01.712613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:01.818897 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:01.818862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:02.178601 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.178570    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:04:02.187474 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.187447    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:04:02.191504 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.191479    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 19:04:02.213783 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.213760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:04:02.221751 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.221730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:04:02.236002 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.235970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:02.336930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.336895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:02.423996 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:02.423854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrzk6" event={"ID":"70e1619c-bd2b-40ef-a1d5-91954ea233dd","Type":"ContainerStarted","Data":"6e03705bc9857eade660592863ecad7f8b189515a134337827b6c0c83974a6d2"}
Jul 08 19:04:03.449803 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:03.449762    2563 scope.go:117] "RemoveContainer" containerID="56e732c5592ccca8d22ca5f12ed04370691cf7a9beb0d91a4b805c7a843612e0"
Jul 08 19:04:03.967428 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:03.967392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:04.434179 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.434153    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:04:04.434375 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.434255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-m929j" event={"ID":"8d1cd19b-8af5-479c-8bbd-47402f98e466","Type":"ContainerStarted","Data":"58fd5143cd4978cfc2f5b476756781866ca997196f66becb3c47b1f8e0c9ab34"}
Jul 08 19:04:04.434591 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.434560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:04:04.436423 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.436395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrzk6" event={"ID":"70e1619c-bd2b-40ef-a1d5-91954ea233dd","Type":"ContainerStarted","Data":"9fcd33a5b806f7a445d7d1d010268215c546f828484909d2811999a6792655f9"}
Jul 08 19:04:04.456264 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.456208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-m929j" podStartSLOduration=43.175931039 podStartE2EDuration="45.456189135s" totalImagesPullingTime="2.280258096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:04.45449122 +0000 UTC m=+303.098511951" watchObservedRunningTime="2026-07-08 19:04:04.456189135 +0000 UTC m=+303.100209866"
Jul 08 19:04:04.477867 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.477781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wrzk6" podStartSLOduration=32.872778263 podStartE2EDuration="40.477768331s" totalImagesPullingTime="7.604990068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:03:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:04.477442342 +0000 UTC m=+303.121463073" watchObservedRunningTime="2026-07-08 19:04:04.477768331 +0000 UTC m=+303.121789062"
Jul 08 19:04:04.516608 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.516582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-m929j"
Jul 08 19:04:04.705325 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.705294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:04.803462 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.803431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:04.902874 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:04.902844    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:05.113951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:05.113852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:05.381899 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:05.381791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:05.483510 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:05.483482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:05.785199 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:05.785152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:05.889570 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:05.889533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:05.994476 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:05.994448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.100131 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.100014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.205201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.205165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.228392 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.228361    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vjknm"]
Jul 08 19:04:06.231979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.231959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.235862 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.235841    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 19:04:06.236011 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.235979    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 19:04:06.236125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.235853    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 19:04:06.237320 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.237303    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 19:04:06.237426 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.237374    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 19:04:06.237426 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.237405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s95pz\""
Jul 08 19:04:06.237537 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.237526    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 19:04:06.320281 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.320255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.351970 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.351890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-wtmp\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352230 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352344 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-root\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352344 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-textfile\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352454 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352512 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-tls\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352565 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9fda5a-6055-41a1-bf07-2e546728b400-metrics-client-ca\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352565 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-sys\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.352687 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.352578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gllj\" (UniqueName: \"kubernetes.io/projected/cc9fda5a-6055-41a1-bf07-2e546728b400-kube-api-access-9gllj\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.431815 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.431785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.453210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9fda5a-6055-41a1-bf07-2e546728b400-metrics-client-ca\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-sys\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gllj\" (UniqueName: \"kubernetes.io/projected/cc9fda5a-6055-41a1-bf07-2e546728b400-kube-api-access-9gllj\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-wtmp\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453301    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-sys\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-root\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-textfile\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453782 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453782 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-wtmp\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453782 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-tls\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.453782 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:04:06.453599    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 08 19:04:06.453782 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:04:06.453667    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-tls podName:cc9fda5a-6055-41a1-bf07-2e546728b400 nodeName:}" failed. No retries permitted until 2026-07-08 19:04:06.953648863 +0000 UTC m=+305.597669577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-tls") pod "node-exporter-vjknm" (UID: "cc9fda5a-6055-41a1-bf07-2e546728b400") : secret "node-exporter-tls" not found
Jul 08 19:04:06.454035 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.453944    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc9fda5a-6055-41a1-bf07-2e546728b400-root\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.454160 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.454138    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc9fda5a-6055-41a1-bf07-2e546728b400-metrics-client-ca\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.454308 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.454235    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-textfile\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.454483 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.454464    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.455926 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.455909    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.463569 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.463548    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gllj\" (UniqueName: \"kubernetes.io/projected/cc9fda5a-6055-41a1-bf07-2e546728b400-kube-api-access-9gllj\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.607125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.607044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.687220 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.687183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.885862 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.885779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:06.958483 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.958450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-tls\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:06.960826 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:06.960802    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc9fda5a-6055-41a1-bf07-2e546728b400-node-exporter-tls\") pod \"node-exporter-vjknm\" (UID: \"cc9fda5a-6055-41a1-bf07-2e546728b400\") " pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:07.142201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.142112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjknm"
Jul 08 19:04:07.154042 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:04:07.153557    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9fda5a_6055_41a1_bf07_2e546728b400.slice/crio-60cb74ab4cfdcc1d6ee622fd9cbd3b6460108df7ba5b3ae546ab1ec9b27ad9a8 WatchSource:0}: Error finding container 60cb74ab4cfdcc1d6ee622fd9cbd3b6460108df7ba5b3ae546ab1ec9b27ad9a8: Status 404 returned error can't find the container with id 60cb74ab4cfdcc1d6ee622fd9cbd3b6460108df7ba5b3ae546ab1ec9b27ad9a8
Jul 08 19:04:07.156060 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.155663    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 19:04:07.184030 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.184005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:07.284418 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.284386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:07.447306 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.447212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjknm" event={"ID":"cc9fda5a-6055-41a1-bf07-2e546728b400","Type":"ContainerStarted","Data":"60cb74ab4cfdcc1d6ee622fd9cbd3b6460108df7ba5b3ae546ab1ec9b27ad9a8"}
Jul 08 19:04:07.485107 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.485076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:07.585228 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.585193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:07.809665 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.809612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:07.912464 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:07.912434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:08.455257 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:08.455225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjknm" event={"ID":"cc9fda5a-6055-41a1-bf07-2e546728b400","Type":"ContainerStarted","Data":"9f9afbc3dadb84f36f444cfe341ef8434ef689961422fcf0e7fcf7e782fd3836"}
Jul 08 19:04:09.161018 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:09.160976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:09.249361 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:09.249318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:09.334511 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:09.334474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:09.457863 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:09.457776    2563 generic.go:356] "Generic (PLEG): container finished" podID="cc9fda5a-6055-41a1-bf07-2e546728b400" containerID="9f9afbc3dadb84f36f444cfe341ef8434ef689961422fcf0e7fcf7e782fd3836" exitCode=0
Jul 08 19:04:09.457863 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:09.457839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjknm" event={"ID":"cc9fda5a-6055-41a1-bf07-2e546728b400","Type":"ContainerDied","Data":"9f9afbc3dadb84f36f444cfe341ef8434ef689961422fcf0e7fcf7e782fd3836"}
Jul 08 19:04:09.761689 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:09.761656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:10.413560 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.413524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:10.463333 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.463301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjknm" event={"ID":"cc9fda5a-6055-41a1-bf07-2e546728b400","Type":"ContainerStarted","Data":"a60504fd2de8e7d8e24999c7a8b30871ec19f22c9dd19a9413de5cf8c3f391a9"}
Jul 08 19:04:10.463484 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.463355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjknm" event={"ID":"cc9fda5a-6055-41a1-bf07-2e546728b400","Type":"ContainerStarted","Data":"80082fb415a4306cd721ace165834b0ed1d9f1994b0afa467c5ca8e6f2543da2"}
Jul 08 19:04:10.492012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.491967    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vjknm" podStartSLOduration=3.254608147 podStartE2EDuration="4.491951453s" totalImagesPullingTime="1.173984566s" totalInitContainerRuntime="63.35874ms" isStatefulPod=true podCreationTimestamp="2026-07-08 19:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:10.490008061 +0000 UTC m=+309.134028793" watchObservedRunningTime="2026-07-08 19:04:10.491951453 +0000 UTC m=+309.135972183"
Jul 08 19:04:10.497030 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.497004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:10.767920 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.767859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:10.866113 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.866081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:10.956987 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:10.956957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:11.021118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:11.021036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:11.259231 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:11.259198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:11.360185 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:11.360101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:12.450977 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:12.450950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:04:12.544044 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:12.544008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:13.169495 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:13.169444    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" podUID="4753956b-81f7-4f49-b651-0424b0191f2c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 08 19:04:14.392861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.392830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fd4f585c9-5ftj8"
Jul 08 19:04:14.537136 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.537102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:14.622864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.622830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:14.626457 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.626431    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84746868fc-zjlbc"]
Jul 08 19:04:14.629493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.629475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.632583 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.632556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-659rr\""
Jul 08 19:04:14.632700 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.632605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 19:04:14.632700 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.632605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 19:04:14.632923 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.632908    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 19:04:14.632971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.632920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 19:04:14.634169 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.634153    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 19:04:14.634240 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.634206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 19:04:14.634316 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.634297    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 19:04:14.640197 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.640176    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84746868fc-zjlbc"]
Jul 08 19:04:14.719538 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.719450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-serving-cert\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.719538 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.719498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-oauth-serving-cert\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.719760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.719553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6ks\" (UniqueName: \"kubernetes.io/projected/1b09ac5d-ba19-48db-a850-4a1871579f7d-kube-api-access-nh6ks\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.719760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.719647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-oauth-config\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.719760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.719680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-config\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.719760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.719711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-service-ca\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.820336 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.820296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-service-ca\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.820493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.820354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-serving-cert\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.820493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.820389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-oauth-serving-cert\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.820493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.820413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6ks\" (UniqueName: \"kubernetes.io/projected/1b09ac5d-ba19-48db-a850-4a1871579f7d-kube-api-access-nh6ks\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.820493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.820472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-oauth-config\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.820752 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.820495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-config\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.821182 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.821158    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-config\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.821303 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.821159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-oauth-serving-cert\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.821303 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.821159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-service-ca\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.822819 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.822799    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-oauth-config\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.822942 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.822926    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-serving-cert\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.837656 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.837612    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6ks\" (UniqueName: \"kubernetes.io/projected/1b09ac5d-ba19-48db-a850-4a1871579f7d-kube-api-access-nh6ks\") pod \"console-84746868fc-zjlbc\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") " pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:14.938841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:14.938804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:15.073944 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:15.073914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84746868fc-zjlbc"]
Jul 08 19:04:15.076311 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:04:15.076281    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b09ac5d_ba19_48db_a850_4a1871579f7d.slice/crio-02c7dfa335f53a0fce8d6465260f8887df491da7e0a29802711b049fdcfd56d6 WatchSource:0}: Error finding container 02c7dfa335f53a0fce8d6465260f8887df491da7e0a29802711b049fdcfd56d6: Status 404 returned error can't find the container with id 02c7dfa335f53a0fce8d6465260f8887df491da7e0a29802711b049fdcfd56d6
Jul 08 19:04:15.207940 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:15.207906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:15.308151 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:15.308116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:15.480369 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:15.480335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84746868fc-zjlbc" event={"ID":"1b09ac5d-ba19-48db-a850-4a1871579f7d","Type":"ContainerStarted","Data":"02c7dfa335f53a0fce8d6465260f8887df491da7e0a29802711b049fdcfd56d6"}
Jul 08 19:04:17.818990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:17.818955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:17.918800 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:17.918758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:19.497735 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:19.497696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84746868fc-zjlbc" event={"ID":"1b09ac5d-ba19-48db-a850-4a1871579f7d","Type":"ContainerStarted","Data":"6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53"}
Jul 08 19:04:19.518727 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:19.518681    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84746868fc-zjlbc" podStartSLOduration=1.730410426 podStartE2EDuration="5.518664876s" totalImagesPullingTime="3.78825445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:04:19.517956922 +0000 UTC m=+318.161977653" watchObservedRunningTime="2026-07-08 19:04:19.518664876 +0000 UTC m=+318.162685605"
Jul 08 19:04:21.205009 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:21.204978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:21.271410 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:21.271366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:23.169018 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:23.168980    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" podUID="4753956b-81f7-4f49-b651-0424b0191f2c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 08 19:04:24.129913 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:24.129876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:24.230895 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:24.230865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:24.939523 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:24.939489    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:24.939740 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:24.939545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:24.944374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:24.944349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:25.409085 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:25.409050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:25.508033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:25.507995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:25.521715 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:25.521694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:04:26.081730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:26.081682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:26.161111 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:26.161060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:33.169235 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:33.169196    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" podUID="4753956b-81f7-4f49-b651-0424b0191f2c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 08 19:04:33.169612 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:33.169266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf"
Jul 08 19:04:33.169812 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:33.169780    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"30c8fba654ae46d3fbf87cedd558816b25c9381496a90f33af3d8a90404c0397"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 08 19:04:33.169854 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:33.169844    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" podUID="4753956b-81f7-4f49-b651-0424b0191f2c" containerName="service-proxy" containerID="cri-o://30c8fba654ae46d3fbf87cedd558816b25c9381496a90f33af3d8a90404c0397" gracePeriod=30
Jul 08 19:04:33.545510 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:33.545476    2563 generic.go:356] "Generic (PLEG): container finished" podID="4753956b-81f7-4f49-b651-0424b0191f2c" containerID="30c8fba654ae46d3fbf87cedd558816b25c9381496a90f33af3d8a90404c0397" exitCode=2
Jul 08 19:04:33.545510 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:33.545514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" event={"ID":"4753956b-81f7-4f49-b651-0424b0191f2c","Type":"ContainerDied","Data":"30c8fba654ae46d3fbf87cedd558816b25c9381496a90f33af3d8a90404c0397"}
Jul 08 19:04:33.545720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:33.545549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c878844-z8blf" event={"ID":"4753956b-81f7-4f49-b651-0424b0191f2c","Type":"ContainerStarted","Data":"b2d234fde2038c343a42af08f6b8adb744e8b1f2aa84f26cbc3f77f955f7878e"}
Jul 08 19:04:35.337522 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:35.337489    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84746868fc-zjlbc"]
Jul 08 19:04:36.823547 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:36.823503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:36.921605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:36.921569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:37.612287 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:37.612259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:37.675442 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:37.675406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:43.973859 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:43.973827    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn77p_488e72ac-a322-4fb9-8553-936a24cc6683/serve-healthcheck-canary/0.log"
Jul 08 19:04:55.396866 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:55.396777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:04:55.499921 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:04:55.499893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:00.357770 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.357730    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-84746868fc-zjlbc" podUID="1b09ac5d-ba19-48db-a850-4a1871579f7d" containerName="console" containerID="cri-o://6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53" gracePeriod=15
Jul 08 19:05:00.599918 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.599895    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:05:00.634344 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.634280    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b09ac5d-ba19-48db-a850-4a1871579f7d" containerID="6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53" exitCode=0
Jul 08 19:05:00.634344 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.634312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84746868fc-zjlbc"
Jul 08 19:05:00.634533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.634360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84746868fc-zjlbc" event={"ID":"1b09ac5d-ba19-48db-a850-4a1871579f7d","Type":"ContainerDied","Data":"6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53"}
Jul 08 19:05:00.634533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.634418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84746868fc-zjlbc" event={"ID":"1b09ac5d-ba19-48db-a850-4a1871579f7d","Type":"ContainerDied","Data":"02c7dfa335f53a0fce8d6465260f8887df491da7e0a29802711b049fdcfd56d6"}
Jul 08 19:05:00.634533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.634446    2563 scope.go:117] "RemoveContainer" containerID="6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53"
Jul 08 19:05:00.643923 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.643906    2563 scope.go:117] "RemoveContainer" containerID="6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53"
Jul 08 19:05:00.644217 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:05:00.644195    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53\": container with ID starting with 6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53 not found: ID does not exist" containerID="6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53"
Jul 08 19:05:00.644310 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.644224    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53"} err="failed to get container status \"6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53\": rpc error: code = NotFound desc = could not find container \"6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53\": container with ID starting with 6ed29f1e983eca51167fe5404b094a22348ac313c447e0e541a1674a3031aa53 not found: ID does not exist"
Jul 08 19:05:00.674020 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-oauth-serving-cert\") pod \"1b09ac5d-ba19-48db-a850-4a1871579f7d\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") "
Jul 08 19:05:00.674118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-oauth-config\") pod \"1b09ac5d-ba19-48db-a850-4a1871579f7d\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") "
Jul 08 19:05:00.674118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh6ks\" (UniqueName: \"kubernetes.io/projected/1b09ac5d-ba19-48db-a850-4a1871579f7d-kube-api-access-nh6ks\") pod \"1b09ac5d-ba19-48db-a850-4a1871579f7d\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") "
Jul 08 19:05:00.674118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-service-ca\") pod \"1b09ac5d-ba19-48db-a850-4a1871579f7d\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") "
Jul 08 19:05:00.674226 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674125    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-config\") pod \"1b09ac5d-ba19-48db-a850-4a1871579f7d\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") "
Jul 08 19:05:00.674226 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-serving-cert\") pod \"1b09ac5d-ba19-48db-a850-4a1871579f7d\" (UID: \"1b09ac5d-ba19-48db-a850-4a1871579f7d\") "
Jul 08 19:05:00.674484 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674462    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b09ac5d-ba19-48db-a850-4a1871579f7d" (UID: "1b09ac5d-ba19-48db-a850-4a1871579f7d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:05:00.674484 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674473    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b09ac5d-ba19-48db-a850-4a1871579f7d" (UID: "1b09ac5d-ba19-48db-a850-4a1871579f7d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:05:00.674656 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.674532    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-config" (OuterVolumeSpecName: "console-config") pod "1b09ac5d-ba19-48db-a850-4a1871579f7d" (UID: "1b09ac5d-ba19-48db-a850-4a1871579f7d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:05:00.676297 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.676268    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b09ac5d-ba19-48db-a850-4a1871579f7d-kube-api-access-nh6ks" (OuterVolumeSpecName: "kube-api-access-nh6ks") pod "1b09ac5d-ba19-48db-a850-4a1871579f7d" (UID: "1b09ac5d-ba19-48db-a850-4a1871579f7d"). InnerVolumeSpecName "kube-api-access-nh6ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:05:00.676374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.676317    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b09ac5d-ba19-48db-a850-4a1871579f7d" (UID: "1b09ac5d-ba19-48db-a850-4a1871579f7d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:05:00.676374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.676341    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b09ac5d-ba19-48db-a850-4a1871579f7d" (UID: "1b09ac5d-ba19-48db-a850-4a1871579f7d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:05:00.774877 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.774842    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-serving-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:05:00.774877 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.774869    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-oauth-serving-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:05:00.774877 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.774878    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-oauth-config\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:05:00.775071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.774887    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6ks\" (UniqueName: \"kubernetes.io/projected/1b09ac5d-ba19-48db-a850-4a1871579f7d-kube-api-access-nh6ks\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:05:00.775071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.774896    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:05:00.775071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.774905    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b09ac5d-ba19-48db-a850-4a1871579f7d-console-config\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:05:00.956228 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.956192    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84746868fc-zjlbc"]
Jul 08 19:05:00.960790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:00.960767    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84746868fc-zjlbc"]
Jul 08 19:05:02.240877 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:02.240849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:02.341687 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:02.341656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:02.402025 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:02.401983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:02.452936 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:02.452859    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b09ac5d-ba19-48db-a850-4a1871579f7d" path="/var/lib/kubelet/pods/1b09ac5d-ba19-48db-a850-4a1871579f7d/volumes"
Jul 08 19:05:05.171373 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:05.171342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:09.609704 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:09.609671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:09.710696 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:09.710665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:16.095636 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:16.095580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:05:16.097925 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:16.097903    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daca3b10-16da-4559-8498-a0bfec82aad8-metrics-certs\") pod \"network-metrics-daemon-r6pt5\" (UID: \"daca3b10-16da-4559-8498-a0bfec82aad8\") " pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:05:16.355227 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:16.355158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvgdf\""
Jul 08 19:05:16.362680 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:16.362660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6pt5"
Jul 08 19:05:16.491773 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:16.491744    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r6pt5"]
Jul 08 19:05:16.494253 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:05:16.494230    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaca3b10_16da_4559_8498_a0bfec82aad8.slice/crio-17ac43139993bb389e358b6066c91f1040ad4685e8d9256004046e1a9ca4e048 WatchSource:0}: Error finding container 17ac43139993bb389e358b6066c91f1040ad4685e8d9256004046e1a9ca4e048: Status 404 returned error can't find the container with id 17ac43139993bb389e358b6066c91f1040ad4685e8d9256004046e1a9ca4e048
Jul 08 19:05:16.695046 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:16.694963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r6pt5" event={"ID":"daca3b10-16da-4559-8498-a0bfec82aad8","Type":"ContainerStarted","Data":"17ac43139993bb389e358b6066c91f1040ad4685e8d9256004046e1a9ca4e048"}
Jul 08 19:05:17.700041 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:17.700014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r6pt5" event={"ID":"daca3b10-16da-4559-8498-a0bfec82aad8","Type":"ContainerStarted","Data":"c771f8df2738b1dab88d826065e4f7c4061abe969ab22e9d11ce26043e5d55d3"}
Jul 08 19:05:18.705359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:18.705315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r6pt5" event={"ID":"daca3b10-16da-4559-8498-a0bfec82aad8","Type":"ContainerStarted","Data":"bb29b8b2f625ed9fc729d1c3e263ee5e7b336ff36fdf136dd4883d896a9e7c53"}
Jul 08 19:05:18.726709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:18.726663    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r6pt5" podStartSLOduration=375.643266956 podStartE2EDuration="6m16.72664855s" totalImagesPullingTime="1.083381594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:18.725888657 +0000 UTC m=+377.369909389" watchObservedRunningTime="2026-07-08 19:05:18.72664855 +0000 UTC m=+377.370669276"
Jul 08 19:05:18.824678 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:18.824643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:18.887479 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:18.887451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:19.561636 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:19.561591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:19.625175 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:19.625149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:25.457149 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:25.457118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:25.656893 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:25.656860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:35.447502 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.447471    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cd7479d8f-p2wxg"]
Jul 08 19:05:35.447932 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.447732    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b09ac5d-ba19-48db-a850-4a1871579f7d" containerName="console"
Jul 08 19:05:35.447932 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.447744    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b09ac5d-ba19-48db-a850-4a1871579f7d" containerName="console"
Jul 08 19:05:35.447932 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.447810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b09ac5d-ba19-48db-a850-4a1871579f7d" containerName="console"
Jul 08 19:05:35.451586 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.451562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.454735 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.454707    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 19:05:35.456168 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.456144    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 19:05:35.456296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.456181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 19:05:35.456296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.456234    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 19:05:35.456405 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.456312    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 19:05:35.456405 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.456376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-659rr\""
Jul 08 19:05:35.456405 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.456387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 19:05:35.456555 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.456406    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 19:05:35.462080 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.462055    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 19:05:35.463364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.463345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd7479d8f-p2wxg"]
Jul 08 19:05:35.507041 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.507015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:35.538219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.538191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-service-ca\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.538364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.538226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-trusted-ca-bundle\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.538364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.538273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-serving-cert\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.538364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.538322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-oauth-config\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.538364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.538341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-oauth-serving-cert\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.538364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.538358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-config\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.538573 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.538381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdvd\" (UniqueName: \"kubernetes.io/projected/2a9f09f3-45a0-464d-994f-5584fd6c5625-kube-api-access-9hdvd\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.569969 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.569938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:35.639225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.639194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-service-ca\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.639225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.639231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-trusted-ca-bundle\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.639472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.639260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-serving-cert\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.639472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.639285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-oauth-config\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.639472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.639308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-oauth-serving-cert\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.639472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.639420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-config\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.639472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.639460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdvd\" (UniqueName: \"kubernetes.io/projected/2a9f09f3-45a0-464d-994f-5584fd6c5625-kube-api-access-9hdvd\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.640114 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.640092    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-service-ca\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.640269 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.640244    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-trusted-ca-bundle\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.640376 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.640151    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-oauth-serving-cert\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.640376 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.640143    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-config\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.641812 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.641794    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-oauth-config\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.641985 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.641968    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-serving-cert\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.648500 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.648479    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdvd\" (UniqueName: \"kubernetes.io/projected/2a9f09f3-45a0-464d-994f-5584fd6c5625-kube-api-access-9hdvd\") pod \"console-7cd7479d8f-p2wxg\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") " pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.761283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.761251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:35.887808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:35.887730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd7479d8f-p2wxg"]
Jul 08 19:05:35.889832 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:05:35.889806    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9f09f3_45a0_464d_994f_5584fd6c5625.slice/crio-2e3667bcbb3238315fa89f529a34d7234c84f67804817d12d404047a15020b68 WatchSource:0}: Error finding container 2e3667bcbb3238315fa89f529a34d7234c84f67804817d12d404047a15020b68: Status 404 returned error can't find the container with id 2e3667bcbb3238315fa89f529a34d7234c84f67804817d12d404047a15020b68
Jul 08 19:05:36.765662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:36.764798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd7479d8f-p2wxg" event={"ID":"2a9f09f3-45a0-464d-994f-5584fd6c5625","Type":"ContainerStarted","Data":"7d7774bc8b9df1881e3486030928c70938d5d2e985cb52433fa6d857a00f1940"}
Jul 08 19:05:36.765662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:36.764906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd7479d8f-p2wxg" event={"ID":"2a9f09f3-45a0-464d-994f-5584fd6c5625","Type":"ContainerStarted","Data":"2e3667bcbb3238315fa89f529a34d7234c84f67804817d12d404047a15020b68"}
Jul 08 19:05:36.786327 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:36.786275    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cd7479d8f-p2wxg" podStartSLOduration=1.786260837 podStartE2EDuration="1.786260837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:05:36.785602275 +0000 UTC m=+395.429623006" watchObservedRunningTime="2026-07-08 19:05:36.786260837 +0000 UTC m=+395.430281565"
Jul 08 19:05:43.794442 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:43.794407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:43.896461 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:43.896423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:45.762117 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:45.762082    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:45.762577 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:45.762135    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:45.766690 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:45.766666    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:45.797445 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:45.797422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:05:45.956698 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:45.956668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:46.057245 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:46.057161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:05:46.122375 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:05:46.122345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:13.956292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:13.956261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:14.014766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:14.014735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:20.260116 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:20.260081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:40.738212 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:40.738180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:40.797497 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:40.797465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:41.371548 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:41.371517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:41.428483 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:41.428459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:51.851471 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:51.851438    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd7479d8f-p2wxg"]
Jul 08 19:06:51.963731 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:51.963694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:52.064745 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:52.064712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:52.122865 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:52.122793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:52.637282 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:52.637248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:52.737763 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:52.737731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:55.393950 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:55.393913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:55.493321 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:55.493286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:06:55.554731 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:06:55.554704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:07:16.873567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:16.873503    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7cd7479d8f-p2wxg" podUID="2a9f09f3-45a0-464d-994f-5584fd6c5625" containerName="console" containerID="cri-o://7d7774bc8b9df1881e3486030928c70938d5d2e985cb52433fa6d857a00f1940" gracePeriod=15
Jul 08 19:07:17.096055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.096022    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a9f09f3-45a0-464d-994f-5584fd6c5625" containerID="7d7774bc8b9df1881e3486030928c70938d5d2e985cb52433fa6d857a00f1940" exitCode=0
Jul 08 19:07:17.096205 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.096094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd7479d8f-p2wxg" event={"ID":"2a9f09f3-45a0-464d-994f-5584fd6c5625","Type":"ContainerDied","Data":"7d7774bc8b9df1881e3486030928c70938d5d2e985cb52433fa6d857a00f1940"}
Jul 08 19:07:17.112111 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.112087    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:07:17.259666 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.259615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hdvd\" (UniqueName: \"kubernetes.io/projected/2a9f09f3-45a0-464d-994f-5584fd6c5625-kube-api-access-9hdvd\") pod \"2a9f09f3-45a0-464d-994f-5584fd6c5625\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") "
Jul 08 19:07:17.259841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.259683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-config\") pod \"2a9f09f3-45a0-464d-994f-5584fd6c5625\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") "
Jul 08 19:07:17.259841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.259720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-service-ca\") pod \"2a9f09f3-45a0-464d-994f-5584fd6c5625\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") "
Jul 08 19:07:17.259841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.259749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-oauth-config\") pod \"2a9f09f3-45a0-464d-994f-5584fd6c5625\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") "
Jul 08 19:07:17.259841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.259786    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-serving-cert\") pod \"2a9f09f3-45a0-464d-994f-5584fd6c5625\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") "
Jul 08 19:07:17.260053 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.259932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-trusted-ca-bundle\") pod \"2a9f09f3-45a0-464d-994f-5584fd6c5625\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") "
Jul 08 19:07:17.260053 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.259999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-oauth-serving-cert\") pod \"2a9f09f3-45a0-464d-994f-5584fd6c5625\" (UID: \"2a9f09f3-45a0-464d-994f-5584fd6c5625\") "
Jul 08 19:07:17.260190 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.260168    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-service-ca" (OuterVolumeSpecName: "service-ca") pod "2a9f09f3-45a0-464d-994f-5584fd6c5625" (UID: "2a9f09f3-45a0-464d-994f-5584fd6c5625"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:07:17.260254 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.260206    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-config" (OuterVolumeSpecName: "console-config") pod "2a9f09f3-45a0-464d-994f-5584fd6c5625" (UID: "2a9f09f3-45a0-464d-994f-5584fd6c5625"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:07:17.260473 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.260445    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2a9f09f3-45a0-464d-994f-5584fd6c5625" (UID: "2a9f09f3-45a0-464d-994f-5584fd6c5625"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:07:17.260521 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.260469    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2a9f09f3-45a0-464d-994f-5584fd6c5625" (UID: "2a9f09f3-45a0-464d-994f-5584fd6c5625"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:07:17.261910 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.261886    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9f09f3-45a0-464d-994f-5584fd6c5625-kube-api-access-9hdvd" (OuterVolumeSpecName: "kube-api-access-9hdvd") pod "2a9f09f3-45a0-464d-994f-5584fd6c5625" (UID: "2a9f09f3-45a0-464d-994f-5584fd6c5625"). InnerVolumeSpecName "kube-api-access-9hdvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:07:17.262274 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.262253    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2a9f09f3-45a0-464d-994f-5584fd6c5625" (UID: "2a9f09f3-45a0-464d-994f-5584fd6c5625"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:07:17.262338 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.262269    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2a9f09f3-45a0-464d-994f-5584fd6c5625" (UID: "2a9f09f3-45a0-464d-994f-5584fd6c5625"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:07:17.360765 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.360738    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-oauth-config\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:07:17.360765 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.360761    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-serving-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:07:17.360765 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.360770    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-trusted-ca-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:07:17.360951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.360779    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-oauth-serving-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:07:17.360951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.360787    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdvd\" (UniqueName: \"kubernetes.io/projected/2a9f09f3-45a0-464d-994f-5584fd6c5625-kube-api-access-9hdvd\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:07:17.360951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.360796    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-console-config\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:07:17.360951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:17.360804    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9f09f3-45a0-464d-994f-5584fd6c5625-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:07:18.100539 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:18.100513    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd7479d8f-p2wxg"
Jul 08 19:07:18.100980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:18.100513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd7479d8f-p2wxg" event={"ID":"2a9f09f3-45a0-464d-994f-5584fd6c5625","Type":"ContainerDied","Data":"2e3667bcbb3238315fa89f529a34d7234c84f67804817d12d404047a15020b68"}
Jul 08 19:07:18.100980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:18.100666    2563 scope.go:117] "RemoveContainer" containerID="7d7774bc8b9df1881e3486030928c70938d5d2e985cb52433fa6d857a00f1940"
Jul 08 19:07:18.125189 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:18.125167    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd7479d8f-p2wxg"]
Jul 08 19:07:18.128744 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:18.128723    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cd7479d8f-p2wxg"]
Jul 08 19:07:18.454616 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:18.454545    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a9f09f3-45a0-464d-994f-5584fd6c5625" path="/var/lib/kubelet/pods/2a9f09f3-45a0-464d-994f-5584fd6c5625/volumes"
Jul 08 19:07:40.729533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:40.729490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:07:51.942121 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:51.942086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:07:52.042113 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.042076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:07:52.956542 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.956507    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"]
Jul 08 19:07:52.956914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.956759    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9f09f3-45a0-464d-994f-5584fd6c5625" containerName="console"
Jul 08 19:07:52.956914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.956771    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9f09f3-45a0-464d-994f-5584fd6c5625" containerName="console"
Jul 08 19:07:52.956914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.956819    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9f09f3-45a0-464d-994f-5584fd6c5625" containerName="console"
Jul 08 19:07:52.959724 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.959703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:52.962810 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.962788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 19:07:52.962810 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.962800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g62bm\""
Jul 08 19:07:52.964232 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.964213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 19:07:52.968723 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.968589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"]
Jul 08 19:07:52.997064 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.997033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:52.997184 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.997080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:52.997184 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:52.997134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqpt\" (UniqueName: \"kubernetes.io/projected/26a02838-93c9-4664-ac94-380d775ace79-kube-api-access-6mqpt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.098031 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.097987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.098168 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.098050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.098168 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.098075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqpt\" (UniqueName: \"kubernetes.io/projected/26a02838-93c9-4664-ac94-380d775ace79-kube-api-access-6mqpt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.098449 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.098429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.098485 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.098439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.108600 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.108569    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqpt\" (UniqueName: \"kubernetes.io/projected/26a02838-93c9-4664-ac94-380d775ace79-kube-api-access-6mqpt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.268748 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.268721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:07:53.395477 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:53.395449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"]
Jul 08 19:07:53.397335 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:07:53.397309    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a02838_93c9_4664_ac94_380d775ace79.slice/crio-3aea69ff2d5a4028a352eea3279f8c3e8f3edb362432861ccb0122a6b21eade6 WatchSource:0}: Error finding container 3aea69ff2d5a4028a352eea3279f8c3e8f3edb362432861ccb0122a6b21eade6: Status 404 returned error can't find the container with id 3aea69ff2d5a4028a352eea3279f8c3e8f3edb362432861ccb0122a6b21eade6
Jul 08 19:07:54.222176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:07:54.222137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr" event={"ID":"26a02838-93c9-4664-ac94-380d775ace79","Type":"ContainerStarted","Data":"3aea69ff2d5a4028a352eea3279f8c3e8f3edb362432861ccb0122a6b21eade6"}
Jul 08 19:08:05.262194 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:05.262158    2563 generic.go:356] "Generic (PLEG): container finished" podID="26a02838-93c9-4664-ac94-380d775ace79" containerID="ab1c00e76acc82404498949b7641bcc15165755f09e9f01b13431e1123d0d455" exitCode=0
Jul 08 19:08:05.262554 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:05.262242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr" event={"ID":"26a02838-93c9-4664-ac94-380d775ace79","Type":"ContainerDied","Data":"ab1c00e76acc82404498949b7641bcc15165755f09e9f01b13431e1123d0d455"}
Jul 08 19:08:08.274916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:08.274883    2563 generic.go:356] "Generic (PLEG): container finished" podID="26a02838-93c9-4664-ac94-380d775ace79" containerID="188c4ef4ba105c9a544ab8a53f06d3b22ae44f4bd2dac9a974e1a862db060102" exitCode=0
Jul 08 19:08:08.275387 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:08.274944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr" event={"ID":"26a02838-93c9-4664-ac94-380d775ace79","Type":"ContainerDied","Data":"188c4ef4ba105c9a544ab8a53f06d3b22ae44f4bd2dac9a974e1a862db060102"}
Jul 08 19:08:15.302951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:15.302914    2563 generic.go:356] "Generic (PLEG): container finished" podID="26a02838-93c9-4664-ac94-380d775ace79" containerID="f6ad4ba0dc67d648abbf235e265e839534044b2935c7cfe7b86580652827fc9e" exitCode=0
Jul 08 19:08:15.303380 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:15.302965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr" event={"ID":"26a02838-93c9-4664-ac94-380d775ace79","Type":"ContainerDied","Data":"f6ad4ba0dc67d648abbf235e265e839534044b2935c7cfe7b86580652827fc9e"}
Jul 08 19:08:16.426076 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.426052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:08:16.476372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.476343    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-util\") pod \"26a02838-93c9-4664-ac94-380d775ace79\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") "
Jul 08 19:08:16.476485 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.476414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-bundle\") pod \"26a02838-93c9-4664-ac94-380d775ace79\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") "
Jul 08 19:08:16.476485 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.476437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mqpt\" (UniqueName: \"kubernetes.io/projected/26a02838-93c9-4664-ac94-380d775ace79-kube-api-access-6mqpt\") pod \"26a02838-93c9-4664-ac94-380d775ace79\" (UID: \"26a02838-93c9-4664-ac94-380d775ace79\") "
Jul 08 19:08:16.477037 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.477013    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-bundle" (OuterVolumeSpecName: "bundle") pod "26a02838-93c9-4664-ac94-380d775ace79" (UID: "26a02838-93c9-4664-ac94-380d775ace79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:08:16.478540 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.478512    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a02838-93c9-4664-ac94-380d775ace79-kube-api-access-6mqpt" (OuterVolumeSpecName: "kube-api-access-6mqpt") pod "26a02838-93c9-4664-ac94-380d775ace79" (UID: "26a02838-93c9-4664-ac94-380d775ace79"). InnerVolumeSpecName "kube-api-access-6mqpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:08:16.480407 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.480384    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-util" (OuterVolumeSpecName: "util") pod "26a02838-93c9-4664-ac94-380d775ace79" (UID: "26a02838-93c9-4664-ac94-380d775ace79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:08:16.577368 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.577310    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:16.577368 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.577332    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqpt\" (UniqueName: \"kubernetes.io/projected/26a02838-93c9-4664-ac94-380d775ace79-kube-api-access-6mqpt\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:16.577368 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:16.577341    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/26a02838-93c9-4664-ac94-380d775ace79-util\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:17.311290 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:17.311250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr" event={"ID":"26a02838-93c9-4664-ac94-380d775ace79","Type":"ContainerDied","Data":"3aea69ff2d5a4028a352eea3279f8c3e8f3edb362432861ccb0122a6b21eade6"}
Jul 08 19:08:17.311290 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:17.311269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pjhpr"
Jul 08 19:08:17.311290 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:17.311295    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aea69ff2d5a4028a352eea3279f8c3e8f3edb362432861ccb0122a6b21eade6"
Jul 08 19:08:19.096718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:19.096687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:20.009537 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:20.009489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:21.246155 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246122    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"]
Jul 08 19:08:21.246494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a02838-93c9-4664-ac94-380d775ace79" containerName="util"
Jul 08 19:08:21.246494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246353    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a02838-93c9-4664-ac94-380d775ace79" containerName="util"
Jul 08 19:08:21.246494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246362    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a02838-93c9-4664-ac94-380d775ace79" containerName="extract"
Jul 08 19:08:21.246494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246368    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a02838-93c9-4664-ac94-380d775ace79" containerName="extract"
Jul 08 19:08:21.246494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246381    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a02838-93c9-4664-ac94-380d775ace79" containerName="pull"
Jul 08 19:08:21.246494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246386    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a02838-93c9-4664-ac94-380d775ace79" containerName="pull"
Jul 08 19:08:21.246494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.246425    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a02838-93c9-4664-ac94-380d775ace79" containerName="extract"
Jul 08 19:08:21.287729 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.287704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"]
Jul 08 19:08:21.287851 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.287799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.291946 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.291923    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vpmc7\""
Jul 08 19:08:21.292564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.292542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 08 19:08:21.302815 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.302795    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 08 19:08:21.408956 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.408923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnrb\" (UniqueName: \"kubernetes.io/projected/afeae211-cbf3-4487-b31f-7c31ea1f2e8c-kube-api-access-2vnrb\") pod \"cert-manager-operator-controller-manager-55c444bf86-b2vck\" (UID: \"afeae211-cbf3-4487-b31f-7c31ea1f2e8c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.409118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.408978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afeae211-cbf3-4487-b31f-7c31ea1f2e8c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b2vck\" (UID: \"afeae211-cbf3-4487-b31f-7c31ea1f2e8c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.510372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.510303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnrb\" (UniqueName: \"kubernetes.io/projected/afeae211-cbf3-4487-b31f-7c31ea1f2e8c-kube-api-access-2vnrb\") pod \"cert-manager-operator-controller-manager-55c444bf86-b2vck\" (UID: \"afeae211-cbf3-4487-b31f-7c31ea1f2e8c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.510372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.510347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afeae211-cbf3-4487-b31f-7c31ea1f2e8c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b2vck\" (UID: \"afeae211-cbf3-4487-b31f-7c31ea1f2e8c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.510702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.510686    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afeae211-cbf3-4487-b31f-7c31ea1f2e8c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b2vck\" (UID: \"afeae211-cbf3-4487-b31f-7c31ea1f2e8c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.524134 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.524106    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnrb\" (UniqueName: \"kubernetes.io/projected/afeae211-cbf3-4487-b31f-7c31ea1f2e8c-kube-api-access-2vnrb\") pod \"cert-manager-operator-controller-manager-55c444bf86-b2vck\" (UID: \"afeae211-cbf3-4487-b31f-7c31ea1f2e8c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.597324 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.597296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"
Jul 08 19:08:21.747965 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:21.747933    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck"]
Jul 08 19:08:21.751700 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:08:21.751675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafeae211_cbf3_4487_b31f_7c31ea1f2e8c.slice/crio-fd54b62e73c873399c0703c0bee79f72f5e453e205a8ac3bd0ed1f11030d3837 WatchSource:0}: Error finding container fd54b62e73c873399c0703c0bee79f72f5e453e205a8ac3bd0ed1f11030d3837: Status 404 returned error can't find the container with id fd54b62e73c873399c0703c0bee79f72f5e453e205a8ac3bd0ed1f11030d3837
Jul 08 19:08:22.330243 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:22.330203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck" event={"ID":"afeae211-cbf3-4487-b31f-7c31ea1f2e8c","Type":"ContainerStarted","Data":"fd54b62e73c873399c0703c0bee79f72f5e453e205a8ac3bd0ed1f11030d3837"}
Jul 08 19:08:23.066113 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:23.066081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:23.168096 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:23.168026    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:24.338640 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:24.338598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck" event={"ID":"afeae211-cbf3-4487-b31f-7c31ea1f2e8c","Type":"ContainerStarted","Data":"81263bd2140d4b67fd8efe0e1e98e313d2bdecff09212662334827e3202ed3d4"}
Jul 08 19:08:24.362028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:24.361970    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b2vck" podStartSLOduration=1.180886817 podStartE2EDuration="3.361951054s" totalImagesPullingTime="2.181064237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:08:24.360151778 +0000 UTC m=+563.004172511" watchObservedRunningTime="2026-07-08 19:08:24.361951054 +0000 UTC m=+563.005971787"
Jul 08 19:08:24.865377 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:24.865340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:24.966007 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:24.965972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:25.016150 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.016117    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"]
Jul 08 19:08:25.033942 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.033917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.034086 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.034046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"]
Jul 08 19:08:25.038025 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.037998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g62bm\""
Jul 08 19:08:25.038146 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.038004    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 19:08:25.039614 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.039595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 19:08:25.136760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.136698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/c70a296c-6738-4b9f-a0c6-620749dc0031-kube-api-access-4wq7c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.136760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.136731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.136760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.136753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.237552 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.237525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/c70a296c-6738-4b9f-a0c6-620749dc0031-kube-api-access-4wq7c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.237720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.237557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.237720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.237581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.237931 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.237914    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.237968 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.237954    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.252054 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.252032    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/c70a296c-6738-4b9f-a0c6-620749dc0031-kube-api-access-4wq7c\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.343744 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.343720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:25.482355 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:25.482332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"]
Jul 08 19:08:25.484288 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:08:25.484259    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70a296c_6738_4b9f_a0c6_620749dc0031.slice/crio-25a764fe0970217b8efda77ae8b55f788f1e619f28365715bc659cd49ace8ecf WatchSource:0}: Error finding container 25a764fe0970217b8efda77ae8b55f788f1e619f28365715bc659cd49ace8ecf: Status 404 returned error can't find the container with id 25a764fe0970217b8efda77ae8b55f788f1e619f28365715bc659cd49ace8ecf
Jul 08 19:08:26.346417 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:26.346378    2563 generic.go:356] "Generic (PLEG): container finished" podID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerID="c7b582a6e3d79565192af03cd0cb89454073122cc672f22faa52bb413a982226" exitCode=0
Jul 08 19:08:26.346773 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:26.346454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t" event={"ID":"c70a296c-6738-4b9f-a0c6-620749dc0031","Type":"ContainerDied","Data":"c7b582a6e3d79565192af03cd0cb89454073122cc672f22faa52bb413a982226"}
Jul 08 19:08:26.346773 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:26.346501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t" event={"ID":"c70a296c-6738-4b9f-a0c6-620749dc0031","Type":"ContainerStarted","Data":"25a764fe0970217b8efda77ae8b55f788f1e619f28365715bc659cd49ace8ecf"}
Jul 08 19:08:27.196012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.195982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:27.624289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.624256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:27.984012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.983940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"]
Jul 08 19:08:27.990767 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.990747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:27.995469 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.995447    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5bgbm\""
Jul 08 19:08:27.995650 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.995512    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 08 19:08:27.995650 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.995516    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 08 19:08:27.995959 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:27.995938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"]
Jul 08 19:08:28.060517 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.060486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55476155-6cfb-4b8b-8043-3500479de343-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gnjgm\" (UID: \"55476155-6cfb-4b8b-8043-3500479de343\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:28.060693 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.060553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnbd\" (UniqueName: \"kubernetes.io/projected/55476155-6cfb-4b8b-8043-3500479de343-kube-api-access-5nnbd\") pod \"cert-manager-cainjector-559cdd8c87-gnjgm\" (UID: \"55476155-6cfb-4b8b-8043-3500479de343\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:28.161995 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.161958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55476155-6cfb-4b8b-8043-3500479de343-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gnjgm\" (UID: \"55476155-6cfb-4b8b-8043-3500479de343\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:28.161995 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.162006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnbd\" (UniqueName: \"kubernetes.io/projected/55476155-6cfb-4b8b-8043-3500479de343-kube-api-access-5nnbd\") pod \"cert-manager-cainjector-559cdd8c87-gnjgm\" (UID: \"55476155-6cfb-4b8b-8043-3500479de343\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:28.174536 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.174502    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55476155-6cfb-4b8b-8043-3500479de343-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gnjgm\" (UID: \"55476155-6cfb-4b8b-8043-3500479de343\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:28.174679 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.174537    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnbd\" (UniqueName: \"kubernetes.io/projected/55476155-6cfb-4b8b-8043-3500479de343-kube-api-access-5nnbd\") pod \"cert-manager-cainjector-559cdd8c87-gnjgm\" (UID: \"55476155-6cfb-4b8b-8043-3500479de343\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:28.301267 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.301240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"
Jul 08 19:08:28.487149 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:28.487110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm"]
Jul 08 19:08:28.488996 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:08:28.488974    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55476155_6cfb_4b8b_8043_3500479de343.slice/crio-41ee4019661a10e920f490eb8744ed895c3fb53019d29a13858cbf880f6d1352 WatchSource:0}: Error finding container 41ee4019661a10e920f490eb8744ed895c3fb53019d29a13858cbf880f6d1352: Status 404 returned error can't find the container with id 41ee4019661a10e920f490eb8744ed895c3fb53019d29a13858cbf880f6d1352
Jul 08 19:08:29.359784 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:29.359744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm" event={"ID":"55476155-6cfb-4b8b-8043-3500479de343","Type":"ContainerStarted","Data":"41ee4019661a10e920f490eb8744ed895c3fb53019d29a13858cbf880f6d1352"}
Jul 08 19:08:29.361942 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:29.361913    2563 generic.go:356] "Generic (PLEG): container finished" podID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerID="46fa7aa5a39a79fb558b7e1ef91f61e40a2c9898f256ec71b7c48c09f4d7d513" exitCode=0
Jul 08 19:08:29.362058 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:29.361975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t" event={"ID":"c70a296c-6738-4b9f-a0c6-620749dc0031","Type":"ContainerDied","Data":"46fa7aa5a39a79fb558b7e1ef91f61e40a2c9898f256ec71b7c48c09f4d7d513"}
Jul 08 19:08:29.882043 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:29.882006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:29.982812 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:29.982775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:30.369876 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:30.369841    2563 generic.go:356] "Generic (PLEG): container finished" podID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerID="98a9f97b1084f2e56646a30792b99e461fe0b15ac3f257b0d8c748e46702fe3d" exitCode=0
Jul 08 19:08:30.370328 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:30.369900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t" event={"ID":"c70a296c-6738-4b9f-a0c6-620749dc0031","Type":"ContainerDied","Data":"98a9f97b1084f2e56646a30792b99e461fe0b15ac3f257b0d8c748e46702fe3d"}
Jul 08 19:08:31.375988 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.375942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm" event={"ID":"55476155-6cfb-4b8b-8043-3500479de343","Type":"ContainerStarted","Data":"681d74bd2382640fb574be1f89ba98d80af9106ad928e7389230282f4e0c56a6"}
Jul 08 19:08:31.398510 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.398463    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gnjgm" podStartSLOduration=1.985833504 podStartE2EDuration="4.398447364s" totalImagesPullingTime="2.41261386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:08:31.39631452 +0000 UTC m=+570.040335250" watchObservedRunningTime="2026-07-08 19:08:31.398447364 +0000 UTC m=+570.042468094"
Jul 08 19:08:31.494592 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.494567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:31.591106 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.591073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-bundle\") pod \"c70a296c-6738-4b9f-a0c6-620749dc0031\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") "
Jul 08 19:08:31.591244 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.591138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-util\") pod \"c70a296c-6738-4b9f-a0c6-620749dc0031\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") "
Jul 08 19:08:31.591244 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.591158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/c70a296c-6738-4b9f-a0c6-620749dc0031-kube-api-access-4wq7c\") pod \"c70a296c-6738-4b9f-a0c6-620749dc0031\" (UID: \"c70a296c-6738-4b9f-a0c6-620749dc0031\") "
Jul 08 19:08:31.591467 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.591445    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-bundle" (OuterVolumeSpecName: "bundle") pod "c70a296c-6738-4b9f-a0c6-620749dc0031" (UID: "c70a296c-6738-4b9f-a0c6-620749dc0031"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:08:31.593234 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.593208    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70a296c-6738-4b9f-a0c6-620749dc0031-kube-api-access-4wq7c" (OuterVolumeSpecName: "kube-api-access-4wq7c") pod "c70a296c-6738-4b9f-a0c6-620749dc0031" (UID: "c70a296c-6738-4b9f-a0c6-620749dc0031"). InnerVolumeSpecName "kube-api-access-4wq7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:08:31.595425 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.595404    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-util" (OuterVolumeSpecName: "util") pod "c70a296c-6738-4b9f-a0c6-620749dc0031" (UID: "c70a296c-6738-4b9f-a0c6-620749dc0031"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:08:31.692563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.692485    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:31.692563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.692509    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70a296c-6738-4b9f-a0c6-620749dc0031-util\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:31.692563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:31.692519    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wq7c\" (UniqueName: \"kubernetes.io/projected/c70a296c-6738-4b9f-a0c6-620749dc0031-kube-api-access-4wq7c\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:32.381173 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:32.381142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t"
Jul 08 19:08:32.381173 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:32.381153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frnr6t" event={"ID":"c70a296c-6738-4b9f-a0c6-620749dc0031","Type":"ContainerDied","Data":"25a764fe0970217b8efda77ae8b55f788f1e619f28365715bc659cd49ace8ecf"}
Jul 08 19:08:32.381656 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:32.381195    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a764fe0970217b8efda77ae8b55f788f1e619f28365715bc659cd49ace8ecf"
Jul 08 19:08:34.820857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:34.820826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:40.265327 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:40.265291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:40.369709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:40.369672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:46.905175 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:46.905139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:48.770343 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:48.770308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:49.027575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:49.027494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:08:50.983525 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983490    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"]
Jul 08 19:08:50.983915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983757    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerName="util"
Jul 08 19:08:50.983915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerName="util"
Jul 08 19:08:50.983915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983779    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerName="extract"
Jul 08 19:08:50.983915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983786    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerName="extract"
Jul 08 19:08:50.983915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983801    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerName="pull"
Jul 08 19:08:50.983915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983808    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerName="pull"
Jul 08 19:08:50.983915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.983850    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70a296c-6738-4b9f-a0c6-620749dc0031" containerName="extract"
Jul 08 19:08:50.986074 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.986057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:50.989466 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.989444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 19:08:50.990839 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.990814    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 19:08:50.990950 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.990818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g62bm\""
Jul 08 19:08:50.997790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:50.997770    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"]
Jul 08 19:08:51.132278 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.132240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzp2n\" (UniqueName: \"kubernetes.io/projected/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-kube-api-access-bzp2n\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.132450 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.132292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.132450 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.132353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.233586 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.233498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.233586 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.233576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzp2n\" (UniqueName: \"kubernetes.io/projected/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-kube-api-access-bzp2n\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.233830 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.233597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.233989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.233964    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.234053 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.233978    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.243880 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.243851    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzp2n\" (UniqueName: \"kubernetes.io/projected/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-kube-api-access-bzp2n\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.294832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.294799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:51.419493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.419467    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"]
Jul 08 19:08:51.422108 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:08:51.422084    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd83de14_45f8_4cb8_bf84_0d1fd36f10f2.slice/crio-8521f90856d76910783724f950a8de934e61499dca8cb82c8dc17315b57c2e62 WatchSource:0}: Error finding container 8521f90856d76910783724f950a8de934e61499dca8cb82c8dc17315b57c2e62: Status 404 returned error can't find the container with id 8521f90856d76910783724f950a8de934e61499dca8cb82c8dc17315b57c2e62
Jul 08 19:08:51.455394 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:51.455368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb" event={"ID":"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2","Type":"ContainerStarted","Data":"8521f90856d76910783724f950a8de934e61499dca8cb82c8dc17315b57c2e62"}
Jul 08 19:08:52.462099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:52.462063    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerID="b28584c0ae3e84e4d34c897b6d64cfbcf89ba5be826c73313e7424e61b80c7c2" exitCode=0
Jul 08 19:08:52.462466 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:52.462147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb" event={"ID":"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2","Type":"ContainerDied","Data":"b28584c0ae3e84e4d34c897b6d64cfbcf89ba5be826c73313e7424e61b80c7c2"}
Jul 08 19:08:53.467605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:53.467516    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerID="f90ce7db2c08f7cf2b4b6c5252b3da965cf2b0d5d7f84c6524ddf9af010314b5" exitCode=0
Jul 08 19:08:53.467605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:53.467569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb" event={"ID":"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2","Type":"ContainerDied","Data":"f90ce7db2c08f7cf2b4b6c5252b3da965cf2b0d5d7f84c6524ddf9af010314b5"}
Jul 08 19:08:54.472802 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:54.472768    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerID="71545ef203e053faee1a73e50e7b0cf9f9e00d15c87666411842d7b78066fd91" exitCode=0
Jul 08 19:08:54.473158 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:54.472817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb" event={"ID":"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2","Type":"ContainerDied","Data":"71545ef203e053faee1a73e50e7b0cf9f9e00d15c87666411842d7b78066fd91"}
Jul 08 19:08:55.600577 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.600554    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:55.766728 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.766697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-util\") pod \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") "
Jul 08 19:08:55.766914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.766753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzp2n\" (UniqueName: \"kubernetes.io/projected/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-kube-api-access-bzp2n\") pod \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") "
Jul 08 19:08:55.766914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.766815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-bundle\") pod \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\" (UID: \"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2\") "
Jul 08 19:08:55.767593 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.767566    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-bundle" (OuterVolumeSpecName: "bundle") pod "dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" (UID: "dd83de14-45f8-4cb8-bf84-0d1fd36f10f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:08:55.768973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.768947    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-kube-api-access-bzp2n" (OuterVolumeSpecName: "kube-api-access-bzp2n") pod "dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" (UID: "dd83de14-45f8-4cb8-bf84-0d1fd36f10f2"). InnerVolumeSpecName "kube-api-access-bzp2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:08:55.772717 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.772691    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-util" (OuterVolumeSpecName: "util") pod "dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" (UID: "dd83de14-45f8-4cb8-bf84-0d1fd36f10f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:08:55.867434 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.867406    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:55.867434 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.867430    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-util\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:55.867434 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:55.867439    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzp2n\" (UniqueName: \"kubernetes.io/projected/dd83de14-45f8-4cb8-bf84-0d1fd36f10f2-kube-api-access-bzp2n\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:08:56.482775 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:56.482749    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb"
Jul 08 19:08:56.482775 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:56.482757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb22gb" event={"ID":"dd83de14-45f8-4cb8-bf84-0d1fd36f10f2","Type":"ContainerDied","Data":"8521f90856d76910783724f950a8de934e61499dca8cb82c8dc17315b57c2e62"}
Jul 08 19:08:56.482937 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:56.482802    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8521f90856d76910783724f950a8de934e61499dca8cb82c8dc17315b57c2e62"
Jul 08 19:08:59.953500 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:08:59.953460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:00.058303 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:00.058259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:01.168172 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:01.168139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:01.314063 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:01.314030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:01.714784 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:01.714754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:02.044792 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.044752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:02.204445 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.204422    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:09:02.211550 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.211528    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:09:02.236214 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.236190    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:09:02.243155 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.243137    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:09:02.268502 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.268469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:02.332916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.332854    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"]
Jul 08 19:09:02.333090 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.333079    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerName="pull"
Jul 08 19:09:02.333130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.333092    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerName="pull"
Jul 08 19:09:02.333130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.333105    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerName="util"
Jul 08 19:09:02.333130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.333111    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerName="util"
Jul 08 19:09:02.333130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.333119    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerName="extract"
Jul 08 19:09:02.333130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.333125    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerName="extract"
Jul 08 19:09:02.333273 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.333165    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd83de14-45f8-4cb8-bf84-0d1fd36f10f2" containerName="extract"
Jul 08 19:09:02.338914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.338893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.342840 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.342815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 08 19:09:02.342956 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.342842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 08 19:09:02.342956 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.342935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 08 19:09:02.343345 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.343332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5zlwm\""
Jul 08 19:09:02.343689 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.343677    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 08 19:09:02.367458 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.367430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"]
Jul 08 19:09:02.516167 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.516141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj47f\" (UniqueName: \"kubernetes.io/projected/05f2e1bb-a497-494f-a5e9-2d40189f287b-kube-api-access-fj47f\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.518210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.516174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05f2e1bb-a497-494f-a5e9-2d40189f287b-manifests\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.518210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.516193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f2e1bb-a497-494f-a5e9-2d40189f287b-tmp\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.518210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.516207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f2e1bb-a497-494f-a5e9-2d40189f287b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.518210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.516262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05f2e1bb-a497-494f-a5e9-2d40189f287b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.524083 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.524064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:02.617298 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.617222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj47f\" (UniqueName: \"kubernetes.io/projected/05f2e1bb-a497-494f-a5e9-2d40189f287b-kube-api-access-fj47f\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.617298 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.617253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05f2e1bb-a497-494f-a5e9-2d40189f287b-manifests\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.617298 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.617270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f2e1bb-a497-494f-a5e9-2d40189f287b-tmp\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.617298 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.617287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f2e1bb-a497-494f-a5e9-2d40189f287b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.617598 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.617578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05f2e1bb-a497-494f-a5e9-2d40189f287b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.617757 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.617737    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/05f2e1bb-a497-494f-a5e9-2d40189f287b-manifests\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.619636 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.619599    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05f2e1bb-a497-494f-a5e9-2d40189f287b-tmp\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.619881 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.619860    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f2e1bb-a497-494f-a5e9-2d40189f287b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.619933 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.619919    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/05f2e1bb-a497-494f-a5e9-2d40189f287b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.626139 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.626117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:02.632590 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.632555    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj47f\" (UniqueName: \"kubernetes.io/projected/05f2e1bb-a497-494f-a5e9-2d40189f287b-kube-api-access-fj47f\") pod \"opendatahub-operator-controller-manager-7dd7775959-2c4cn\" (UID: \"05f2e1bb-a497-494f-a5e9-2d40189f287b\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.648285 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.648256    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:02.724893 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.724859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:02.795867 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.793723    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"]
Jul 08 19:09:02.796050 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:09:02.795987    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f2e1bb_a497_494f_a5e9_2d40189f287b.slice/crio-a903f53e51f7276ab5b962204380d60e752e418ec03e392e66790e5e57ec5cc0 WatchSource:0}: Error finding container a903f53e51f7276ab5b962204380d60e752e418ec03e392e66790e5e57ec5cc0: Status 404 returned error can't find the container with id a903f53e51f7276ab5b962204380d60e752e418ec03e392e66790e5e57ec5cc0
Jul 08 19:09:02.825071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.825016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:02.931111 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:02.931002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:03.039394 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:03.039360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:03.511949 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:03.511913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn" event={"ID":"05f2e1bb-a497-494f-a5e9-2d40189f287b","Type":"ContainerStarted","Data":"a903f53e51f7276ab5b962204380d60e752e418ec03e392e66790e5e57ec5cc0"}
Jul 08 19:09:05.351758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:05.351723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:05.578378 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:05.578340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:06.529088 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:06.529051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn" event={"ID":"05f2e1bb-a497-494f-a5e9-2d40189f287b","Type":"ContainerStarted","Data":"3ae1f9be7aea835f7e5b3e170103d44782bd8fbd2878e34ce39aeb1890ee1b57"}
Jul 08 19:09:07.534265 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:07.534228    2563 generic.go:356] "Generic (PLEG): container finished" podID="05f2e1bb-a497-494f-a5e9-2d40189f287b" containerID="3ae1f9be7aea835f7e5b3e170103d44782bd8fbd2878e34ce39aeb1890ee1b57" exitCode=0
Jul 08 19:09:07.534673 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:07.534278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn" event={"ID":"05f2e1bb-a497-494f-a5e9-2d40189f287b","Type":"ContainerDied","Data":"3ae1f9be7aea835f7e5b3e170103d44782bd8fbd2878e34ce39aeb1890ee1b57"}
Jul 08 19:09:07.534766 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:07.534751    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 19:09:08.540084 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:08.540054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn" event={"ID":"05f2e1bb-a497-494f-a5e9-2d40189f287b","Type":"ContainerStarted","Data":"e41f5a5a803094cd99683638bf97caf8e80ac0b4b7a1fdcacb71355257109570"}
Jul 08 19:09:08.540457 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:08.540151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:08.573403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:08.573351    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn" podStartSLOduration=2.105770562 podStartE2EDuration="6.573336266s" totalImagesPullingTime="3.402908835s" totalInitContainerRuntime="1.064656869s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:08.571307313 +0000 UTC m=+607.215328043" watchObservedRunningTime="2026-07-08 19:09:08.573336266 +0000 UTC m=+607.217356996"
Jul 08 19:09:13.883164 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:13.883125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:19.545858 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:19.545825    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd7775959-2c4cn"
Jul 08 19:09:21.407126 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:21.407055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:21.601427 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:21.601396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:22.202161 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.202127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:22.341844 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.341807    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"]
Jul 08 19:09:22.347658 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.347616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.351268 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.351249    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 08 19:09:22.352915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.352894    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 08 19:09:22.352915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.352890    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 08 19:09:22.353057 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.352894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 08 19:09:22.365178 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.365155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f64f61-b641-4fd8-a445-f5bafd042491-tls-certs\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.365272 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.365188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f64f61-b641-4fd8-a445-f5bafd042491-tmp\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.365272 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.365205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf8rj\" (UniqueName: \"kubernetes.io/projected/22f64f61-b641-4fd8-a445-f5bafd042491-kube-api-access-mf8rj\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.372416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.372386    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"]
Jul 08 19:09:22.466508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.466418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f64f61-b641-4fd8-a445-f5bafd042491-tls-certs\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.466508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.466453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f64f61-b641-4fd8-a445-f5bafd042491-tmp\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.466508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.466477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf8rj\" (UniqueName: \"kubernetes.io/projected/22f64f61-b641-4fd8-a445-f5bafd042491-kube-api-access-mf8rj\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.467056 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:09:22.466551    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 08 19:09:22.467056 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:09:22.466637    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f64f61-b641-4fd8-a445-f5bafd042491-tls-certs podName:22f64f61-b641-4fd8-a445-f5bafd042491 nodeName:}" failed. No retries permitted until 2026-07-08 19:09:22.966608992 +0000 UTC m=+621.610629730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/22f64f61-b641-4fd8-a445-f5bafd042491-tls-certs") pod "kube-auth-proxy-85bd9454df-ll5cn" (UID: "22f64f61-b641-4fd8-a445-f5bafd042491") : secret "kube-auth-proxy-tls" not found
Jul 08 19:09:22.468913 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.468890    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f64f61-b641-4fd8-a445-f5bafd042491-tmp\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.486020 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.485988    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf8rj\" (UniqueName: \"kubernetes.io/projected/22f64f61-b641-4fd8-a445-f5bafd042491-kube-api-access-mf8rj\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.558071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.558042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:22.653745 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.653707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:22.763110 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.763075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:22.855170 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.855131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:22.970449 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.970416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f64f61-b641-4fd8-a445-f5bafd042491-tls-certs\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:22.972833 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:22.972803    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22f64f61-b641-4fd8-a445-f5bafd042491-tls-certs\") pod \"kube-auth-proxy-85bd9454df-ll5cn\" (UID: \"22f64f61-b641-4fd8-a445-f5bafd042491\") " pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:23.006872 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.006835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:23.171379 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.171298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:23.220033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.219954    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xmdfl"]
Jul 08 19:09:23.222067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.222051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:23.225318 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.225302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 08 19:09:23.239608 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.239571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xmdfl"]
Jul 08 19:09:23.258077 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.258051    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"
Jul 08 19:09:23.271773 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.271737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkblf\" (UniqueName: \"kubernetes.io/projected/03487c46-ee6c-4f8f-b703-4747db34c9af-kube-api-access-lkblf\") pod \"model-serving-api-54d89bd5fc-xmdfl\" (UID: \"03487c46-ee6c-4f8f-b703-4747db34c9af\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:23.271892 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.271803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03487c46-ee6c-4f8f-b703-4747db34c9af-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmdfl\" (UID: \"03487c46-ee6c-4f8f-b703-4747db34c9af\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:23.312933 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.312891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:23.373204 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.373164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkblf\" (UniqueName: \"kubernetes.io/projected/03487c46-ee6c-4f8f-b703-4747db34c9af-kube-api-access-lkblf\") pod \"model-serving-api-54d89bd5fc-xmdfl\" (UID: \"03487c46-ee6c-4f8f-b703-4747db34c9af\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:23.373351 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.373250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03487c46-ee6c-4f8f-b703-4747db34c9af-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmdfl\" (UID: \"03487c46-ee6c-4f8f-b703-4747db34c9af\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:23.373397 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:09:23.373383    2563 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 08 19:09:23.373468 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:09:23.373457    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03487c46-ee6c-4f8f-b703-4747db34c9af-tls-certs podName:03487c46-ee6c-4f8f-b703-4747db34c9af nodeName:}" failed. No retries permitted until 2026-07-08 19:09:23.873437641 +0000 UTC m=+622.517458364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/03487c46-ee6c-4f8f-b703-4747db34c9af-tls-certs") pod "model-serving-api-54d89bd5fc-xmdfl" (UID: "03487c46-ee6c-4f8f-b703-4747db34c9af") : secret "model-serving-api-tls" not found
Jul 08 19:09:23.396964 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.396939    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkblf\" (UniqueName: \"kubernetes.io/projected/03487c46-ee6c-4f8f-b703-4747db34c9af-kube-api-access-lkblf\") pod \"model-serving-api-54d89bd5fc-xmdfl\" (UID: \"03487c46-ee6c-4f8f-b703-4747db34c9af\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:23.419652 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.419611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn"]
Jul 08 19:09:23.421200 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:09:23.421175    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f64f61_b641_4fd8_a445_f5bafd042491.slice/crio-b7e42c06efe76b19fb364049ade0d92168e075428affe6e7a1012fe2b429c2a4 WatchSource:0}: Error finding container b7e42c06efe76b19fb364049ade0d92168e075428affe6e7a1012fe2b429c2a4: Status 404 returned error can't find the container with id b7e42c06efe76b19fb364049ade0d92168e075428affe6e7a1012fe2b429c2a4
Jul 08 19:09:23.598032 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.597994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn" event={"ID":"22f64f61-b641-4fd8-a445-f5bafd042491","Type":"ContainerStarted","Data":"b7e42c06efe76b19fb364049ade0d92168e075428affe6e7a1012fe2b429c2a4"}
Jul 08 19:09:23.719002 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.718904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:23.877032 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.877001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03487c46-ee6c-4f8f-b703-4747db34c9af-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmdfl\" (UID: \"03487c46-ee6c-4f8f-b703-4747db34c9af\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:23.879348 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:23.879315    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03487c46-ee6c-4f8f-b703-4747db34c9af-tls-certs\") pod \"model-serving-api-54d89bd5fc-xmdfl\" (UID: \"03487c46-ee6c-4f8f-b703-4747db34c9af\") " pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:24.137146 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:24.137112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:24.295877 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:24.295818    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xmdfl"]
Jul 08 19:09:24.298333 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:09:24.298305    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03487c46_ee6c_4f8f_b703_4747db34c9af.slice/crio-82c2352202fc87e817b1c6a0eaf5017541d67f6a90f48b22a6a30c359debd923 WatchSource:0}: Error finding container 82c2352202fc87e817b1c6a0eaf5017541d67f6a90f48b22a6a30c359debd923: Status 404 returned error can't find the container with id 82c2352202fc87e817b1c6a0eaf5017541d67f6a90f48b22a6a30c359debd923
Jul 08 19:09:24.440323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:24.440249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:24.551543 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:24.551508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:24.603363 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:24.603330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl" event={"ID":"03487c46-ee6c-4f8f-b703-4747db34c9af","Type":"ContainerStarted","Data":"82c2352202fc87e817b1c6a0eaf5017541d67f6a90f48b22a6a30c359debd923"}
Jul 08 19:09:25.595984 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:25.595940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:27.617264 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:27.617227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn" event={"ID":"22f64f61-b641-4fd8-a445-f5bafd042491","Type":"ContainerStarted","Data":"c3efe6d0833bf21ae38fbf39e8b34073f523d1aa6bcd9ec56475fcc40d121d25"}
Jul 08 19:09:27.643694 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:27.643636    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85bd9454df-ll5cn" podStartSLOduration=2.322724034 podStartE2EDuration="5.643601858s" totalImagesPullingTime="3.320877824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:27.642699877 +0000 UTC m=+626.286720610" watchObservedRunningTime="2026-07-08 19:09:27.643601858 +0000 UTC m=+626.287622590"
Jul 08 19:09:28.623095 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:28.623060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl" event={"ID":"03487c46-ee6c-4f8f-b703-4747db34c9af","Type":"ContainerStarted","Data":"4f7f3468e1c48f7fde40c6c27731b9a857ef2211569c69e0abaef9827ae57f90"}
Jul 08 19:09:28.623486 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:28.623210    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:28.643874 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:28.643804    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl" podStartSLOduration=2.254636676 podStartE2EDuration="5.643790153s" totalImagesPullingTime="3.389153477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:28.642811209 +0000 UTC m=+627.286831950" watchObservedRunningTime="2026-07-08 19:09:28.643790153 +0000 UTC m=+627.287810882"
Jul 08 19:09:30.488844 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.488811    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"]
Jul 08 19:09:30.490798 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.490783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.502224 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.502204    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 08 19:09:30.502446 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.502432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gj98c\""
Jul 08 19:09:30.502872 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.502858    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 08 19:09:30.503636 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.503612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 08 19:09:30.504829 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.504816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 08 19:09:30.525569 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.525546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"]
Jul 08 19:09:30.530260 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.530240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33312d-7f53-452b-9489-1eb07e098525-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.530379 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.530285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.530379 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.530305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b33312d-7f53-452b-9489-1eb07e098525-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.530494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.530390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.530494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.530443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b33312d-7f53-452b-9489-1eb07e098525-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.530494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.530489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.530674 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.530542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875v6\" (UniqueName: \"kubernetes.io/projected/9b33312d-7f53-452b-9489-1eb07e098525-kube-api-access-875v6\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.631549 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.631519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33312d-7f53-452b-9489-1eb07e098525-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.631761 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.631556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.631761 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.631575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b33312d-7f53-452b-9489-1eb07e098525-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.631761 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.631684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.631761 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.631716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b33312d-7f53-452b-9489-1eb07e098525-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.631761 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.631746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.632031 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.631792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875v6\" (UniqueName: \"kubernetes.io/projected/9b33312d-7f53-452b-9489-1eb07e098525-kube-api-access-875v6\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.632341 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.632308    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b33312d-7f53-452b-9489-1eb07e098525-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.633959 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.633937    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b33312d-7f53-452b-9489-1eb07e098525-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.634108 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.634089    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.634439 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.634421    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.634520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.634439    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b33312d-7f53-452b-9489-1eb07e098525-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.655319 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.655292    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b33312d-7f53-452b-9489-1eb07e098525-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.661458 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.661436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-875v6\" (UniqueName: \"kubernetes.io/projected/9b33312d-7f53-452b-9489-1eb07e098525-kube-api-access-875v6\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnpm\" (UID: \"9b33312d-7f53-452b-9489-1eb07e098525\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.799717 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.799618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:30.937578 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:30.937544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"]
Jul 08 19:09:30.940430 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:09:30.940389    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b33312d_7f53_452b_9489_1eb07e098525.slice/crio-fa25d1bfd61de701e6f3396a3e32a53e4409cc6314c21df681302d8014f48b04 WatchSource:0}: Error finding container fa25d1bfd61de701e6f3396a3e32a53e4409cc6314c21df681302d8014f48b04: Status 404 returned error can't find the container with id fa25d1bfd61de701e6f3396a3e32a53e4409cc6314c21df681302d8014f48b04
Jul 08 19:09:31.640314 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:31.640258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm" event={"ID":"9b33312d-7f53-452b-9489-1eb07e098525","Type":"ContainerStarted","Data":"fa25d1bfd61de701e6f3396a3e32a53e4409cc6314c21df681302d8014f48b04"}
Jul 08 19:09:33.162794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:33.162760    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:09:33.163061 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:33.162823    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:09:33.650156 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:33.650123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm" event={"ID":"9b33312d-7f53-452b-9489-1eb07e098525","Type":"ContainerStarted","Data":"d117639f9818354dbc38abc0dc224e1d94f9e55de97b81ff1a54c4898bc62011"}
Jul 08 19:09:33.650347 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:33.650218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:33.680914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:33.680862    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm" podStartSLOduration=1.46146657 podStartE2EDuration="3.680847066s" totalImagesPullingTime="2.219380496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:33.678467476 +0000 UTC m=+632.322488205" watchObservedRunningTime="2026-07-08 19:09:33.680847066 +0000 UTC m=+632.324867797"
Jul 08 19:09:34.267318 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.267283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:34.431319 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.431291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:34.581558 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.581468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:34.655924 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.655893    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-dxnpm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 08 19:09:34.656064 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.655959    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm" podUID="9b33312d-7f53-452b-9489-1eb07e098525" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 19:09:34.758979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.758947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:34.813597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.813559    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"]
Jul 08 19:09:34.817086 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.817062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.821798 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.821773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qdg7s\""
Jul 08 19:09:34.841684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.841591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"]
Jul 08 19:09:34.867025 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.866991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6w6\" (UniqueName: \"kubernetes.io/projected/d3c9f26c-cd40-43b9-bca0-daab0c641500-kube-api-access-zd6w6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f26c-cd40-43b9-bca0-daab0c641500-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.867465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.867323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.892225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.892192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:34.967968 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.967932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.967968 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.967974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968166 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6w6\" (UniqueName: \"kubernetes.io/projected/d3c9f26c-cd40-43b9-bca0-daab0c641500-kube-api-access-zd6w6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968166 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968166 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968166 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f26c-cd40-43b9-bca0-daab0c641500-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968570 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968465    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968570 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968490    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968751 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968721    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.968897 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.968840    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.969088 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.969069    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c9f26c-cd40-43b9-bca0-daab0c641500-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.970654 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.970634    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:34.970654 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:34.970645    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:35.005985 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.005954    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c9f26c-cd40-43b9-bca0-daab0c641500-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:35.006090 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.005992    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6w6\" (UniqueName: \"kubernetes.io/projected/d3c9f26c-cd40-43b9-bca0-daab0c641500-kube-api-access-zd6w6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v9lvc\" (UID: \"d3c9f26c-cd40-43b9-bca0-daab0c641500\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:35.089118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.089089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:35.129140 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.129075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:35.339125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.339092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:35.349681 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:09:35.349655    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c9f26c_cd40_43b9_bca0_daab0c641500.slice/crio-762fd59dda3e1ef83bba162d08d78ec3107250c27246b44c5664dd5bb5a5e0f0 WatchSource:0}: Error finding container 762fd59dda3e1ef83bba162d08d78ec3107250c27246b44c5664dd5bb5a5e0f0: Status 404 returned error can't find the container with id 762fd59dda3e1ef83bba162d08d78ec3107250c27246b44c5664dd5bb5a5e0f0
Jul 08 19:09:35.355667 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.355643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"]
Jul 08 19:09:35.492209 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.492178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:35.660497 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.660462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc" event={"ID":"d3c9f26c-cd40-43b9-bca0-daab0c641500","Type":"ContainerStarted","Data":"762fd59dda3e1ef83bba162d08d78ec3107250c27246b44c5664dd5bb5a5e0f0"}
Jul 08 19:09:35.923987 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:35.923894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:36.078671 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:36.078636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:36.297267 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:36.297225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:37.656005 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:37.655977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnpm"
Jul 08 19:09:38.004616 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:38.004585    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:09:38.004698 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:38.004672    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:09:38.004737 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:38.004702    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:09:38.676420 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:38.676383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc" event={"ID":"d3c9f26c-cd40-43b9-bca0-daab0c641500","Type":"ContainerStarted","Data":"323247430d3fd61be6adbf5620d1f282e25c698d3f31c814855fe29c5e096bdd"}
Jul 08 19:09:38.742318 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:38.742265    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc" podStartSLOduration=2.091383739 podStartE2EDuration="4.742249088s" totalImagesPullingTime="2.650865349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:38.740688834 +0000 UTC m=+637.384709563" watchObservedRunningTime="2026-07-08 19:09:38.742249088 +0000 UTC m=+637.386269874"
Jul 08 19:09:39.130157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:39.130120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:39.134797 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:39.134773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:39.631692 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:39.631663    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xmdfl"
Jul 08 19:09:39.681202 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:39.681174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:39.682312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:39.682289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v9lvc"
Jul 08 19:09:42.453175 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:42.453144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:09:42.522260 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:09:42.522229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:22.845420 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:22.845384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:22.948809 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:22.948774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:23.346476 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:23.346438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:38.674428 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.674397    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"]
Jul 08 19:10:38.677779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.677762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.680726 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.680701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 08 19:10:38.680876 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.680757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 08 19:10:38.680876 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.680808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lhrps\""
Jul 08 19:10:38.688995 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.688958    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"]
Jul 08 19:10:38.793312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.793276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.793508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.793320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.793508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.793356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcw9w\" (UniqueName: \"kubernetes.io/projected/90afc36d-4dcd-48af-8676-8084cda5a3c1-kube-api-access-bcw9w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.894686 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.894640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.894864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.894702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.894864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.894760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcw9w\" (UniqueName: \"kubernetes.io/projected/90afc36d-4dcd-48af-8676-8084cda5a3c1-kube-api-access-bcw9w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.894976 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.894958    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.895078 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.895057    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.907691 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.907663    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcw9w\" (UniqueName: \"kubernetes.io/projected/90afc36d-4dcd-48af-8676-8084cda5a3c1-kube-api-access-bcw9w\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:38.992508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:38.992474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:39.137935 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.126028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"]
Jul 08 19:10:39.149004 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:10:39.148973    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90afc36d_4dcd_48af_8676_8084cda5a3c1.slice/crio-8eba1ac0dd50315055d2df50534d2e6f52776a69994e9f3ba4a14700c8ab1026 WatchSource:0}: Error finding container 8eba1ac0dd50315055d2df50534d2e6f52776a69994e9f3ba4a14700c8ab1026: Status 404 returned error can't find the container with id 8eba1ac0dd50315055d2df50534d2e6f52776a69994e9f3ba4a14700c8ab1026
Jul 08 19:10:39.312974 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.312940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"]
Jul 08 19:10:39.316283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.316263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.327852 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.327828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"]
Jul 08 19:10:39.399044 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.399008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.399204 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.399056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.399204 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.399142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67xv\" (UniqueName: \"kubernetes.io/projected/97ced5c2-b241-48ce-981f-3a93ad25e215-kube-api-access-n67xv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.500476 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.500392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67xv\" (UniqueName: \"kubernetes.io/projected/97ced5c2-b241-48ce-981f-3a93ad25e215-kube-api-access-n67xv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.500476 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.500454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.500719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.500484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.500854 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.500836    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.500903 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.500865    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.512603 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.512582    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67xv\" (UniqueName: \"kubernetes.io/projected/97ced5c2-b241-48ce-981f-3a93ad25e215-kube-api-access-n67xv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.625026 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.624997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:39.681195 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.681167    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"]
Jul 08 19:10:39.684783 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.684756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.694836 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.694812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"]
Jul 08 19:10:39.760043 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.760016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"]
Jul 08 19:10:39.761762 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:10:39.761739    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ced5c2_b241_48ce_981f_3a93ad25e215.slice/crio-2c40262408fa2d2c08a8784001885411dd4692c60bad6573e3856946061726ba WatchSource:0}: Error finding container 2c40262408fa2d2c08a8784001885411dd4692c60bad6573e3856946061726ba: Status 404 returned error can't find the container with id 2c40262408fa2d2c08a8784001885411dd4692c60bad6573e3856946061726ba
Jul 08 19:10:39.802617 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.802593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.802732 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.802644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68sbl\" (UniqueName: \"kubernetes.io/projected/764cbca2-713e-46ae-ba62-be468308f6e3-kube-api-access-68sbl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.802732 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.802711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.903502 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.903472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.903679 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.903556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.903679 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.903577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68sbl\" (UniqueName: \"kubernetes.io/projected/764cbca2-713e-46ae-ba62-be468308f6e3-kube-api-access-68sbl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.903944 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.903919    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.904012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.903929    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.913452 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.913429    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68sbl\" (UniqueName: \"kubernetes.io/projected/764cbca2-713e-46ae-ba62-be468308f6e3-kube-api-access-68sbl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:39.931751 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.931721    2563 generic.go:356] "Generic (PLEG): container finished" podID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerID="33273ad51b934e6e0638f57ffcac4c4db16852e4d0ecc4dd6c5da6ae4e21c6fa" exitCode=0
Jul 08 19:10:39.931869 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.931749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r" event={"ID":"90afc36d-4dcd-48af-8676-8084cda5a3c1","Type":"ContainerDied","Data":"33273ad51b934e6e0638f57ffcac4c4db16852e4d0ecc4dd6c5da6ae4e21c6fa"}
Jul 08 19:10:39.931869 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.931799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r" event={"ID":"90afc36d-4dcd-48af-8676-8084cda5a3c1","Type":"ContainerStarted","Data":"8eba1ac0dd50315055d2df50534d2e6f52776a69994e9f3ba4a14700c8ab1026"}
Jul 08 19:10:39.933542 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.933521    2563 generic.go:356] "Generic (PLEG): container finished" podID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerID="f25870e8e15a235a0c27bddd9254658ce4b0993256cbbc5c06ad713db2310fa6" exitCode=0
Jul 08 19:10:39.933651 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.933547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q" event={"ID":"97ced5c2-b241-48ce-981f-3a93ad25e215","Type":"ContainerDied","Data":"f25870e8e15a235a0c27bddd9254658ce4b0993256cbbc5c06ad713db2310fa6"}
Jul 08 19:10:39.933651 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.933577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q" event={"ID":"97ced5c2-b241-48ce-981f-3a93ad25e215","Type":"ContainerStarted","Data":"2c40262408fa2d2c08a8784001885411dd4692c60bad6573e3856946061726ba"}
Jul 08 19:10:39.994747 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:39.994716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:40.077772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.077743    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"]
Jul 08 19:10:40.081635 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.081596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.092335 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.092307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"]
Jul 08 19:10:40.131604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.131577    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"]
Jul 08 19:10:40.133068 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:10:40.133042    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764cbca2_713e_46ae_ba62_be468308f6e3.slice/crio-83036b4b1a0f96caf27c7cb8a6143e5b0c153d288a8a3c5c56b3e5dc3ea3a0ab WatchSource:0}: Error finding container 83036b4b1a0f96caf27c7cb8a6143e5b0c153d288a8a3c5c56b3e5dc3ea3a0ab: Status 404 returned error can't find the container with id 83036b4b1a0f96caf27c7cb8a6143e5b0c153d288a8a3c5c56b3e5dc3ea3a0ab
Jul 08 19:10:40.207015 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.206960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.207200 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.207089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgdq\" (UniqueName: \"kubernetes.io/projected/e78667e8-b469-4030-a5a4-16e3ba683528-kube-api-access-5rgdq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.207200 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.207123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.308105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.308018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.308105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.308069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgdq\" (UniqueName: \"kubernetes.io/projected/e78667e8-b469-4030-a5a4-16e3ba683528-kube-api-access-5rgdq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.308105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.308089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.308397 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.308378    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.308481 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.308465    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.318320 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.318299    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgdq\" (UniqueName: \"kubernetes.io/projected/e78667e8-b469-4030-a5a4-16e3ba683528-kube-api-access-5rgdq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.393341 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.393303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:40.520916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.520891    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"]
Jul 08 19:10:40.522452 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:10:40.522428    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78667e8_b469_4030_a5a4_16e3ba683528.slice/crio-7ded42ae28e5c6bf2e44977037301932aea4835470618f575e7c6fccd945a30b WatchSource:0}: Error finding container 7ded42ae28e5c6bf2e44977037301932aea4835470618f575e7c6fccd945a30b: Status 404 returned error can't find the container with id 7ded42ae28e5c6bf2e44977037301932aea4835470618f575e7c6fccd945a30b
Jul 08 19:10:40.939276 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.939248    2563 generic.go:356] "Generic (PLEG): container finished" podID="e78667e8-b469-4030-a5a4-16e3ba683528" containerID="e19c286916fd4171f775d587e13a8d75a67101422f4f0fc15fb8a20f21ac9aba" exitCode=0
Jul 08 19:10:40.939569 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.939321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj" event={"ID":"e78667e8-b469-4030-a5a4-16e3ba683528","Type":"ContainerDied","Data":"e19c286916fd4171f775d587e13a8d75a67101422f4f0fc15fb8a20f21ac9aba"}
Jul 08 19:10:40.939569 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.939368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj" event={"ID":"e78667e8-b469-4030-a5a4-16e3ba683528","Type":"ContainerStarted","Data":"7ded42ae28e5c6bf2e44977037301932aea4835470618f575e7c6fccd945a30b"}
Jul 08 19:10:40.940986 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.940968    2563 generic.go:356] "Generic (PLEG): container finished" podID="764cbca2-713e-46ae-ba62-be468308f6e3" containerID="9c962fce95d3bddd1f232ea13b00491a77d384fba13e0b4fa12b5ad699960f3f" exitCode=0
Jul 08 19:10:40.941071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.941013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d" event={"ID":"764cbca2-713e-46ae-ba62-be468308f6e3","Type":"ContainerDied","Data":"9c962fce95d3bddd1f232ea13b00491a77d384fba13e0b4fa12b5ad699960f3f"}
Jul 08 19:10:40.941071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:40.941045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d" event={"ID":"764cbca2-713e-46ae-ba62-be468308f6e3","Type":"ContainerStarted","Data":"83036b4b1a0f96caf27c7cb8a6143e5b0c153d288a8a3c5c56b3e5dc3ea3a0ab"}
Jul 08 19:10:41.949914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:41.949883    2563 generic.go:356] "Generic (PLEG): container finished" podID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerID="8aea7cc6e1200f7a110c0553c0d16a78be52a48b6ac0778491f200b86396ea70" exitCode=0
Jul 08 19:10:41.950256 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:41.949969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q" event={"ID":"97ced5c2-b241-48ce-981f-3a93ad25e215","Type":"ContainerDied","Data":"8aea7cc6e1200f7a110c0553c0d16a78be52a48b6ac0778491f200b86396ea70"}
Jul 08 19:10:41.951895 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:41.951872    2563 generic.go:356] "Generic (PLEG): container finished" podID="764cbca2-713e-46ae-ba62-be468308f6e3" containerID="190d9186bb4b07387754b6871f8d681a870cbe242058b7c562956e1372efe24e" exitCode=0
Jul 08 19:10:41.952005 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:41.951955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d" event={"ID":"764cbca2-713e-46ae-ba62-be468308f6e3","Type":"ContainerDied","Data":"190d9186bb4b07387754b6871f8d681a870cbe242058b7c562956e1372efe24e"}
Jul 08 19:10:41.953930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:41.953908    2563 generic.go:356] "Generic (PLEG): container finished" podID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerID="cd7bc8cd443925ee3ce49c69568dbe3b723a678e25967cf9acb20d75c4dd5c18" exitCode=0
Jul 08 19:10:41.954026 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:41.953934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r" event={"ID":"90afc36d-4dcd-48af-8676-8084cda5a3c1","Type":"ContainerDied","Data":"cd7bc8cd443925ee3ce49c69568dbe3b723a678e25967cf9acb20d75c4dd5c18"}
Jul 08 19:10:42.960664 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.960608    2563 generic.go:356] "Generic (PLEG): container finished" podID="764cbca2-713e-46ae-ba62-be468308f6e3" containerID="29e9fc76516200c502754ec0f7343b2325c3dbb0cf407cbe0105424381767094" exitCode=0
Jul 08 19:10:42.961057 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.960697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d" event={"ID":"764cbca2-713e-46ae-ba62-be468308f6e3","Type":"ContainerDied","Data":"29e9fc76516200c502754ec0f7343b2325c3dbb0cf407cbe0105424381767094"}
Jul 08 19:10:42.962741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.962715    2563 generic.go:356] "Generic (PLEG): container finished" podID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerID="435fc24c794d2fdbf803dc86bbcd1f6553898f2b35e7b579554d8fe97df43f24" exitCode=0
Jul 08 19:10:42.962876 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.962800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r" event={"ID":"90afc36d-4dcd-48af-8676-8084cda5a3c1","Type":"ContainerDied","Data":"435fc24c794d2fdbf803dc86bbcd1f6553898f2b35e7b579554d8fe97df43f24"}
Jul 08 19:10:42.964365 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.964344    2563 generic.go:356] "Generic (PLEG): container finished" podID="e78667e8-b469-4030-a5a4-16e3ba683528" containerID="8ad51036efed103a0b9c2f9649a6b6a6bf8820d6c9b790798ba885b59e98df19" exitCode=0
Jul 08 19:10:42.964471 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.964424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj" event={"ID":"e78667e8-b469-4030-a5a4-16e3ba683528","Type":"ContainerDied","Data":"8ad51036efed103a0b9c2f9649a6b6a6bf8820d6c9b790798ba885b59e98df19"}
Jul 08 19:10:42.966507 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.966487    2563 generic.go:356] "Generic (PLEG): container finished" podID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerID="541cc6e9d955a12dcda92e316e07ecdf8852ca95a5df912eb96da7cd4a378b17" exitCode=0
Jul 08 19:10:42.966596 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:42.966540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q" event={"ID":"97ced5c2-b241-48ce-981f-3a93ad25e215","Type":"ContainerDied","Data":"541cc6e9d955a12dcda92e316e07ecdf8852ca95a5df912eb96da7cd4a378b17"}
Jul 08 19:10:43.972869 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:43.972836    2563 generic.go:356] "Generic (PLEG): container finished" podID="e78667e8-b469-4030-a5a4-16e3ba683528" containerID="1dddf166a28f9388958d39a99a36b65e270d63655e0b8cfd9338f77a32248861" exitCode=0
Jul 08 19:10:43.973319 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:43.972922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj" event={"ID":"e78667e8-b469-4030-a5a4-16e3ba683528","Type":"ContainerDied","Data":"1dddf166a28f9388958d39a99a36b65e270d63655e0b8cfd9338f77a32248861"}
Jul 08 19:10:44.134760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.134736    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:44.137794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.137735    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:44.141369 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.141342    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:44.244534 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244508    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-util\") pod \"97ced5c2-b241-48ce-981f-3a93ad25e215\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") "
Jul 08 19:10:44.244718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-util\") pod \"764cbca2-713e-46ae-ba62-be468308f6e3\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") "
Jul 08 19:10:44.244718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244581    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-bundle\") pod \"764cbca2-713e-46ae-ba62-be468308f6e3\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") "
Jul 08 19:10:44.244718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-bundle\") pod \"97ced5c2-b241-48ce-981f-3a93ad25e215\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") "
Jul 08 19:10:44.244718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-bundle\") pod \"90afc36d-4dcd-48af-8676-8084cda5a3c1\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") "
Jul 08 19:10:44.244926 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244722    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n67xv\" (UniqueName: \"kubernetes.io/projected/97ced5c2-b241-48ce-981f-3a93ad25e215-kube-api-access-n67xv\") pod \"97ced5c2-b241-48ce-981f-3a93ad25e215\" (UID: \"97ced5c2-b241-48ce-981f-3a93ad25e215\") "
Jul 08 19:10:44.244926 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-util\") pod \"90afc36d-4dcd-48af-8676-8084cda5a3c1\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") "
Jul 08 19:10:44.244926 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcw9w\" (UniqueName: \"kubernetes.io/projected/90afc36d-4dcd-48af-8676-8084cda5a3c1-kube-api-access-bcw9w\") pod \"90afc36d-4dcd-48af-8676-8084cda5a3c1\" (UID: \"90afc36d-4dcd-48af-8676-8084cda5a3c1\") "
Jul 08 19:10:44.244926 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.244864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68sbl\" (UniqueName: \"kubernetes.io/projected/764cbca2-713e-46ae-ba62-be468308f6e3-kube-api-access-68sbl\") pod \"764cbca2-713e-46ae-ba62-be468308f6e3\" (UID: \"764cbca2-713e-46ae-ba62-be468308f6e3\") "
Jul 08 19:10:44.245264 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.245231    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-bundle" (OuterVolumeSpecName: "bundle") pod "764cbca2-713e-46ae-ba62-be468308f6e3" (UID: "764cbca2-713e-46ae-ba62-be468308f6e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:44.245351 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.245319    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-bundle" (OuterVolumeSpecName: "bundle") pod "90afc36d-4dcd-48af-8676-8084cda5a3c1" (UID: "90afc36d-4dcd-48af-8676-8084cda5a3c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:44.245416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.245387    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-bundle" (OuterVolumeSpecName: "bundle") pod "97ced5c2-b241-48ce-981f-3a93ad25e215" (UID: "97ced5c2-b241-48ce-981f-3a93ad25e215"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:44.247303 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.247280    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764cbca2-713e-46ae-ba62-be468308f6e3-kube-api-access-68sbl" (OuterVolumeSpecName: "kube-api-access-68sbl") pod "764cbca2-713e-46ae-ba62-be468308f6e3" (UID: "764cbca2-713e-46ae-ba62-be468308f6e3"). InnerVolumeSpecName "kube-api-access-68sbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:10:44.247399 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.247314    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ced5c2-b241-48ce-981f-3a93ad25e215-kube-api-access-n67xv" (OuterVolumeSpecName: "kube-api-access-n67xv") pod "97ced5c2-b241-48ce-981f-3a93ad25e215" (UID: "97ced5c2-b241-48ce-981f-3a93ad25e215"). InnerVolumeSpecName "kube-api-access-n67xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:10:44.247466 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.247446    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90afc36d-4dcd-48af-8676-8084cda5a3c1-kube-api-access-bcw9w" (OuterVolumeSpecName: "kube-api-access-bcw9w") pod "90afc36d-4dcd-48af-8676-8084cda5a3c1" (UID: "90afc36d-4dcd-48af-8676-8084cda5a3c1"). InnerVolumeSpecName "kube-api-access-bcw9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:10:44.250857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.250837    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-util" (OuterVolumeSpecName: "util") pod "97ced5c2-b241-48ce-981f-3a93ad25e215" (UID: "97ced5c2-b241-48ce-981f-3a93ad25e215"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:44.254020 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.253992    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-util" (OuterVolumeSpecName: "util") pod "90afc36d-4dcd-48af-8676-8084cda5a3c1" (UID: "90afc36d-4dcd-48af-8676-8084cda5a3c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:44.254072 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.254035    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-util" (OuterVolumeSpecName: "util") pod "764cbca2-713e-46ae-ba62-be468308f6e3" (UID: "764cbca2-713e-46ae-ba62-be468308f6e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:44.346504 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346466    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcw9w\" (UniqueName: \"kubernetes.io/projected/90afc36d-4dcd-48af-8676-8084cda5a3c1-kube-api-access-bcw9w\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346504 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346496    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68sbl\" (UniqueName: \"kubernetes.io/projected/764cbca2-713e-46ae-ba62-be468308f6e3-kube-api-access-68sbl\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346504 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346506    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-util\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346504 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346514    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-util\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346522    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/764cbca2-713e-46ae-ba62-be468308f6e3-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346530    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97ced5c2-b241-48ce-981f-3a93ad25e215-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346561    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346570    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n67xv\" (UniqueName: \"kubernetes.io/projected/97ced5c2-b241-48ce-981f-3a93ad25e215-kube-api-access-n67xv\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.346794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.346580    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90afc36d-4dcd-48af-8676-8084cda5a3c1-util\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:44.979440 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.979411    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q"
Jul 08 19:10:44.979440 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.979421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q" event={"ID":"97ced5c2-b241-48ce-981f-3a93ad25e215","Type":"ContainerDied","Data":"2c40262408fa2d2c08a8784001885411dd4692c60bad6573e3856946061726ba"}
Jul 08 19:10:44.979933 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.979463    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c40262408fa2d2c08a8784001885411dd4692c60bad6573e3856946061726ba"
Jul 08 19:10:44.981294 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.981274    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d"
Jul 08 19:10:44.981294 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.981277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d" event={"ID":"764cbca2-713e-46ae-ba62-be468308f6e3","Type":"ContainerDied","Data":"83036b4b1a0f96caf27c7cb8a6143e5b0c153d288a8a3c5c56b3e5dc3ea3a0ab"}
Jul 08 19:10:44.981491 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.981322    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83036b4b1a0f96caf27c7cb8a6143e5b0c153d288a8a3c5c56b3e5dc3ea3a0ab"
Jul 08 19:10:44.983506 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.983485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r" event={"ID":"90afc36d-4dcd-48af-8676-8084cda5a3c1","Type":"ContainerDied","Data":"8eba1ac0dd50315055d2df50534d2e6f52776a69994e9f3ba4a14700c8ab1026"}
Jul 08 19:10:44.983610 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.983523    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eba1ac0dd50315055d2df50534d2e6f52776a69994e9f3ba4a14700c8ab1026"
Jul 08 19:10:44.983687 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:44.983656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r"
Jul 08 19:10:45.102916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.102895    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:45.253730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.253702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-bundle\") pod \"e78667e8-b469-4030-a5a4-16e3ba683528\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") "
Jul 08 19:10:45.253890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.253767    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-util\") pod \"e78667e8-b469-4030-a5a4-16e3ba683528\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") "
Jul 08 19:10:45.253890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.253807    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rgdq\" (UniqueName: \"kubernetes.io/projected/e78667e8-b469-4030-a5a4-16e3ba683528-kube-api-access-5rgdq\") pod \"e78667e8-b469-4030-a5a4-16e3ba683528\" (UID: \"e78667e8-b469-4030-a5a4-16e3ba683528\") "
Jul 08 19:10:45.254196 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.254171    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-bundle" (OuterVolumeSpecName: "bundle") pod "e78667e8-b469-4030-a5a4-16e3ba683528" (UID: "e78667e8-b469-4030-a5a4-16e3ba683528"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:45.255957 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.255928    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78667e8-b469-4030-a5a4-16e3ba683528-kube-api-access-5rgdq" (OuterVolumeSpecName: "kube-api-access-5rgdq") pod "e78667e8-b469-4030-a5a4-16e3ba683528" (UID: "e78667e8-b469-4030-a5a4-16e3ba683528"). InnerVolumeSpecName "kube-api-access-5rgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:10:45.261508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.261471    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-util" (OuterVolumeSpecName: "util") pod "e78667e8-b469-4030-a5a4-16e3ba683528" (UID: "e78667e8-b469-4030-a5a4-16e3ba683528"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:10:45.354472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.354439    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-util\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:45.354472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.354469    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgdq\" (UniqueName: \"kubernetes.io/projected/e78667e8-b469-4030-a5a4-16e3ba683528-kube-api-access-5rgdq\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:45.354472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.354478    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e78667e8-b469-4030-a5a4-16e3ba683528-bundle\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:10:45.990292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.990260    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj"
Jul 08 19:10:45.990292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.990283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj" event={"ID":"e78667e8-b469-4030-a5a4-16e3ba683528","Type":"ContainerDied","Data":"7ded42ae28e5c6bf2e44977037301932aea4835470618f575e7c6fccd945a30b"}
Jul 08 19:10:45.990835 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:45.990327    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ded42ae28e5c6bf2e44977037301932aea4835470618f575e7c6fccd945a30b"
Jul 08 19:10:49.950978 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.950941    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7778bdfcc-v8llj"]
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951206    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78667e8-b469-4030-a5a4-16e3ba683528" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951217    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78667e8-b469-4030-a5a4-16e3ba683528" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951230    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951235    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951248    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951255    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764cbca2-713e-46ae-ba62-be468308f6e3" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951260    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="764cbca2-713e-46ae-ba62-be468308f6e3" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951267    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951272    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951283    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951289    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951303    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951308    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951313    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764cbca2-713e-46ae-ba62-be468308f6e3" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951318    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="764cbca2-713e-46ae-ba62-be468308f6e3" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951327    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764cbca2-713e-46ae-ba62-be468308f6e3" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951332    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="764cbca2-713e-46ae-ba62-be468308f6e3" containerName="util"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951339    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78667e8-b469-4030-a5a4-16e3ba683528" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951344    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78667e8-b469-4030-a5a4-16e3ba683528" containerName="pull"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951349    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78667e8-b469-4030-a5a4-16e3ba683528" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951353    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78667e8-b469-4030-a5a4-16e3ba683528" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951397    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78667e8-b469-4030-a5a4-16e3ba683528" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951405    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ced5c2-b241-48ce-981f-3a93ad25e215" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951410    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="764cbca2-713e-46ae-ba62-be468308f6e3" containerName="extract"
Jul 08 19:10:49.951463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.951418    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90afc36d-4dcd-48af-8676-8084cda5a3c1" containerName="extract"
Jul 08 19:10:49.954712 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.954685    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:49.957853 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.957828    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 19:10:49.959280 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.959263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 19:10:49.959360 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.959307    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 19:10:49.959360 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.959349    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 19:10:49.959464 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.959448    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 19:10:49.959585 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.959562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-659rr\""
Jul 08 19:10:49.959840 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.959825    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 19:10:49.959891 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.959826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 19:10:49.964487 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.964456    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 19:10:49.967457 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:49.967435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7778bdfcc-v8llj"]
Jul 08 19:10:50.043160 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.043127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:50.095012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.094977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-serving-cert\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.095174 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.095025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-oauth-serving-cert\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.095174 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.095080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-oauth-config\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.095174 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.095107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-trusted-ca-bundle\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.095174 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.095140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-config\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.095174 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.095159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4798v\" (UniqueName: \"kubernetes.io/projected/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-kube-api-access-4798v\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.095345 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.095183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-service-ca\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.105772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.105740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:50.196580 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.196539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-oauth-serving-cert\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.196580 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.196588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-oauth-config\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.196857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.196611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-trusted-ca-bundle\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.196857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.196659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-config\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.196857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.196685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4798v\" (UniqueName: \"kubernetes.io/projected/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-kube-api-access-4798v\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.196857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.196719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-service-ca\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.196857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.196801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-serving-cert\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.197483 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.197445    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-config\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.197601 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.197514    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-oauth-serving-cert\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.197601 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.197537    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-service-ca\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.197810 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.197790    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-trusted-ca-bundle\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.199416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.199391    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-serving-cert\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.199532 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.199436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-console-oauth-config\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.206598 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.206541    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4798v\" (UniqueName: \"kubernetes.io/projected/ca93bd6a-fd2f-494b-bb3c-6341f71efac9-kube-api-access-4798v\") pod \"console-7778bdfcc-v8llj\" (UID: \"ca93bd6a-fd2f-494b-bb3c-6341f71efac9\") " pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.264774 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.264738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:10:50.396416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:50.396388    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7778bdfcc-v8llj"]
Jul 08 19:10:50.398162 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:10:50.398135    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca93bd6a_fd2f_494b_bb3c_6341f71efac9.slice/crio-60bbaaeda8507e88e48d8d7d3dcdb7a02667ed864165dc17584cf8a18313c0d2 WatchSource:0}: Error finding container 60bbaaeda8507e88e48d8d7d3dcdb7a02667ed864165dc17584cf8a18313c0d2: Status 404 returned error can't find the container with id 60bbaaeda8507e88e48d8d7d3dcdb7a02667ed864165dc17584cf8a18313c0d2
Jul 08 19:10:51.015381 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:51.015344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7778bdfcc-v8llj" event={"ID":"ca93bd6a-fd2f-494b-bb3c-6341f71efac9","Type":"ContainerStarted","Data":"09e05b65eeca0a76da32190b1bcb7b5a788891e352765bf3c6b528ae7d55373f"}
Jul 08 19:10:51.015769 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:51.015399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7778bdfcc-v8llj" event={"ID":"ca93bd6a-fd2f-494b-bb3c-6341f71efac9","Type":"ContainerStarted","Data":"60bbaaeda8507e88e48d8d7d3dcdb7a02667ed864165dc17584cf8a18313c0d2"}
Jul 08 19:10:51.043915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:51.043861    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7778bdfcc-v8llj" podStartSLOduration=2.043846325 podStartE2EDuration="2.043846325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:10:51.041483456 +0000 UTC m=+709.685504188" watchObservedRunningTime="2026-07-08 19:10:51.043846325 +0000 UTC m=+709.687867118"
Jul 08 19:10:52.347661 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:52.347601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:56.346177 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:56.346093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:58.222369 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.222333    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"]
Jul 08 19:10:58.225418 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.225401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"
Jul 08 19:10:58.228370 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.228348    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 08 19:10:58.229777 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.229755    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 08 19:10:58.229777 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.229755    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 08 19:10:58.229937 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.229815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jf94h\""
Jul 08 19:10:58.237911 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.237891    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"]
Jul 08 19:10:58.363952 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.363917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5mz\" (UniqueName: \"kubernetes.io/projected/318ba7b6-0cb9-401f-b981-2a3b7339003d-kube-api-access-7j5mz\") pod \"dns-operator-controller-manager-864dd78c4c-z4qqc\" (UID: \"318ba7b6-0cb9-401f-b981-2a3b7339003d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"
Jul 08 19:10:58.464584 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.464546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5mz\" (UniqueName: \"kubernetes.io/projected/318ba7b6-0cb9-401f-b981-2a3b7339003d-kube-api-access-7j5mz\") pod \"dns-operator-controller-manager-864dd78c4c-z4qqc\" (UID: \"318ba7b6-0cb9-401f-b981-2a3b7339003d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"
Jul 08 19:10:58.476907 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.476842    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5mz\" (UniqueName: \"kubernetes.io/projected/318ba7b6-0cb9-401f-b981-2a3b7339003d-kube-api-access-7j5mz\") pod \"dns-operator-controller-manager-864dd78c4c-z4qqc\" (UID: \"318ba7b6-0cb9-401f-b981-2a3b7339003d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"
Jul 08 19:10:58.536475 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.536436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"
Jul 08 19:10:58.669456 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.669432    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"]
Jul 08 19:10:58.671480 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:10:58.671453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318ba7b6_0cb9_401f_b981_2a3b7339003d.slice/crio-2527049f0d52e6073e776e0267c351e75e8785cee20f70c2670dd02f525448da WatchSource:0}: Error finding container 2527049f0d52e6073e776e0267c351e75e8785cee20f70c2670dd02f525448da: Status 404 returned error can't find the container with id 2527049f0d52e6073e776e0267c351e75e8785cee20f70c2670dd02f525448da
Jul 08 19:10:58.951710 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:58.951680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:10:59.052762 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:10:59.052728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc" event={"ID":"318ba7b6-0cb9-401f-b981-2a3b7339003d","Type":"ContainerStarted","Data":"2527049f0d52e6073e776e0267c351e75e8785cee20f70c2670dd02f525448da"}
Jul 08 19:11:00.265573 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:00.265538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:11:00.266044 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:00.265670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:11:00.270956 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:00.270933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:11:00.603057 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:00.602982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:01.069053 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:01.069024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7778bdfcc-v8llj"
Jul 08 19:11:01.277806 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:01.277760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:01.383665 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:01.383574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:01.482130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:01.482098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:02.022704 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:02.022670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:02.072591 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:02.072549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc" event={"ID":"318ba7b6-0cb9-401f-b981-2a3b7339003d","Type":"ContainerStarted","Data":"b9d92e872887ef7575ee70d8a2d8d297f8c764b094e3c18004593ef25b53751d"}
Jul 08 19:11:02.072923 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:02.072787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"
Jul 08 19:11:02.110159 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:02.110107    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc" podStartSLOduration=1.279101484 podStartE2EDuration="4.110091974s" totalImagesPullingTime="2.83099049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:02.107855082 +0000 UTC m=+720.751875813" watchObservedRunningTime="2026-07-08 19:11:02.110091974 +0000 UTC m=+720.754112704"
Jul 08 19:11:03.208412 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.208383    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"]
Jul 08 19:11:03.211614 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.211596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.215342 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.215321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-szlr2\""
Jul 08 19:11:03.229610 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.229587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"]
Jul 08 19:11:03.309311 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.309279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjdl\" (UniqueName: \"kubernetes.io/projected/1bbddd27-0253-4781-9ffd-3e7098d6068b-kube-api-access-mfjdl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.309472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.309348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbddd27-0253-4781-9ffd-3e7098d6068b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.410321 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.410277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbddd27-0253-4781-9ffd-3e7098d6068b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.410464 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.410343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjdl\" (UniqueName: \"kubernetes.io/projected/1bbddd27-0253-4781-9ffd-3e7098d6068b-kube-api-access-mfjdl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.410683 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.410664    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbddd27-0253-4781-9ffd-3e7098d6068b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.420482 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.420457    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjdl\" (UniqueName: \"kubernetes.io/projected/1bbddd27-0253-4781-9ffd-3e7098d6068b-kube-api-access-mfjdl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.520605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.520577    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:03.567555 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.567524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:03.656375 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.656352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"]
Jul 08 19:11:03.658131 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:11:03.658105    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbddd27_0253_4781_9ffd_3e7098d6068b.slice/crio-a6ff19c1372941624779881d69e992e7fc85b9b47b7973369dd34f437b30633d WatchSource:0}: Error finding container a6ff19c1372941624779881d69e992e7fc85b9b47b7973369dd34f437b30633d: Status 404 returned error can't find the container with id a6ff19c1372941624779881d69e992e7fc85b9b47b7973369dd34f437b30633d
Jul 08 19:11:03.792533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:03.792457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:04.082800 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:04.082720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692" event={"ID":"1bbddd27-0253-4781-9ffd-3e7098d6068b","Type":"ContainerStarted","Data":"a6ff19c1372941624779881d69e992e7fc85b9b47b7973369dd34f437b30633d"}
Jul 08 19:11:04.232224 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:04.232182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:06.228444 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:06.228389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:08.782159 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:08.782122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:08.882152 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:08.882117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:10.570546 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:10.570505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:10.702605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:10.702571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:12.126091 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:12.126054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692" event={"ID":"1bbddd27-0253-4781-9ffd-3e7098d6068b","Type":"ContainerStarted","Data":"67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf"}
Jul 08 19:11:12.126538 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:12.126197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:12.150386 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:12.150334    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692" podStartSLOduration=1.199253865 podStartE2EDuration="9.150318683s" totalImagesPullingTime="7.951064818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:12.147716075 +0000 UTC m=+730.791736806" watchObservedRunningTime="2026-07-08 19:11:12.150318683 +0000 UTC m=+730.794339413"
Jul 08 19:11:13.079393 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:13.079368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4qqc"
Jul 08 19:11:14.646107 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:14.646059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:14.798634 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:14.798577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:15.745931 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:15.745898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:15.845598 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:15.845566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:23.134370 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:23.134340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:24.175453 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.175417    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"]
Jul 08 19:11:24.179856 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.179838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.193213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.193191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"]
Jul 08 19:11:24.195534 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.195515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:24.296113 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.296079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d98aeb19-35b1-4900-af99-1b05cb3450c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.296275 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.296171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxfq\" (UniqueName: \"kubernetes.io/projected/d98aeb19-35b1-4900-af99-1b05cb3450c4-kube-api-access-5sxfq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.297131 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.297108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:24.396777 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.396749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxfq\" (UniqueName: \"kubernetes.io/projected/d98aeb19-35b1-4900-af99-1b05cb3450c4-kube-api-access-5sxfq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.396937 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.396787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d98aeb19-35b1-4900-af99-1b05cb3450c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.397159 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.397140    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d98aeb19-35b1-4900-af99-1b05cb3450c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.407286 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.407264    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxfq\" (UniqueName: \"kubernetes.io/projected/d98aeb19-35b1-4900-af99-1b05cb3450c4-kube-api-access-5sxfq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.491010 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.490982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:24.632873 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.632850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"]
Jul 08 19:11:24.634445 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:11:24.634417    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98aeb19_35b1_4900_af99_1b05cb3450c4.slice/crio-0fa86f751994ae722f3990c74c65771f655c426e9f9982dafe2b9219c0bf0467 WatchSource:0}: Error finding container 0fa86f751994ae722f3990c74c65771f655c426e9f9982dafe2b9219c0bf0467: Status 404 returned error can't find the container with id 0fa86f751994ae722f3990c74c65771f655c426e9f9982dafe2b9219c0bf0467
Jul 08 19:11:24.918959 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.918878    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"]
Jul 08 19:11:24.926034 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.925995    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"]
Jul 08 19:11:24.946486 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.946445    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"]
Jul 08 19:11:24.946729 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.946697    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" containerName="manager" containerID="cri-o://67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf" gracePeriod=2
Jul 08 19:11:24.953685 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.953653    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"]
Jul 08 19:11:24.954277 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.954087    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" containerName="manager"
Jul 08 19:11:24.954277 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.954108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" containerName="manager"
Jul 08 19:11:24.954277 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.954218    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" containerName="manager"
Jul 08 19:11:24.957617 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.957599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:24.959859 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.959835    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"]
Jul 08 19:11:24.966950 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:24.966802    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1bbddd27-0253-4781-9ffd-3e7098d6068b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-jp692\": pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:24.970308 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:24.970281    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:24.971051 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.970987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"]
Jul 08 19:11:24.982232 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.982213    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"]
Jul 08 19:11:24.982479 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.982468    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" containerName="manager"
Jul 08 19:11:24.982520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.982480    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" containerName="manager"
Jul 08 19:11:24.982553 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.982536    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" containerName="manager"
Jul 08 19:11:24.985356 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:24.985340    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.001734 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.001705    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"]
Jul 08 19:11:25.018692 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.018654    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:25.102829 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.102795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lt2k\" (UniqueName: \"kubernetes.io/projected/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-kube-api-access-6lt2k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qh8j\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:25.102982 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.102843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qh8j\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:25.102982 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.102928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhtq\" (UniqueName: \"kubernetes.io/projected/c996b6b9-b03d-422d-9c22-713736a39756-kube-api-access-6nhtq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cc2n\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.102982 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.102955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c996b6b9-b03d-422d-9c22-713736a39756-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cc2n\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.115299 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.115270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:25.191434 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.191411    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:25.192173 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.192155    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k" container="manager"
Jul 08 19:11:25.193386 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.193362    2563 generic.go:356] "Generic (PLEG): container finished" podID="1bbddd27-0253-4781-9ffd-3e7098d6068b" containerID="67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf" exitCode=0
Jul 08 19:11:25.193513 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.193396    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:25.193513 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.193424    2563 scope.go:117] "RemoveContainer" containerID="67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf"
Jul 08 19:11:25.194299 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.194272    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:25.196830 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.196801    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:25.199239 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.199215    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:25.202391 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.202375    2563 scope.go:117] "RemoveContainer" containerID="67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf"
Jul 08 19:11:25.202670 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.202652    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf\": container with ID starting with 67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf not found: ID does not exist" containerID="67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf"
Jul 08 19:11:25.202741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.202681    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf"} err="failed to get container status \"67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf\": rpc error: code = NotFound desc = could not find container \"67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf\": container with ID starting with 67e04b5870bf77f5e5dd275ad1765a0bdc9d2ecee279c93c4083dbe68b769bbf not found: ID does not exist"
Jul 08 19:11:25.204400 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.204376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lt2k\" (UniqueName: \"kubernetes.io/projected/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-kube-api-access-6lt2k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qh8j\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:25.204563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.204539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qh8j\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:25.204685 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.204605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhtq\" (UniqueName: \"kubernetes.io/projected/c996b6b9-b03d-422d-9c22-713736a39756-kube-api-access-6nhtq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cc2n\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.204685 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.204649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c996b6b9-b03d-422d-9c22-713736a39756-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cc2n\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.205126 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.205011    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c996b6b9-b03d-422d-9c22-713736a39756-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cc2n\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.205330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.205290    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qh8j\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:25.216279 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.216258    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lt2k\" (UniqueName: \"kubernetes.io/projected/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-kube-api-access-6lt2k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qh8j\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:25.216380 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.216332    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhtq\" (UniqueName: \"kubernetes.io/projected/c996b6b9-b03d-422d-9c22-713736a39756-kube-api-access-6nhtq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cc2n\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.216474 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.216392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:25.305678 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.305646    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbddd27-0253-4781-9ffd-3e7098d6068b-extensions-socket-volume\") pod \"1bbddd27-0253-4781-9ffd-3e7098d6068b\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") "
Jul 08 19:11:25.305810 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.305687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfjdl\" (UniqueName: \"kubernetes.io/projected/1bbddd27-0253-4781-9ffd-3e7098d6068b-kube-api-access-mfjdl\") pod \"1bbddd27-0253-4781-9ffd-3e7098d6068b\" (UID: \"1bbddd27-0253-4781-9ffd-3e7098d6068b\") "
Jul 08 19:11:25.306088 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.306064    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bbddd27-0253-4781-9ffd-3e7098d6068b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1bbddd27-0253-4781-9ffd-3e7098d6068b" (UID: "1bbddd27-0253-4781-9ffd-3e7098d6068b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:11:25.307602 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.307582    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbddd27-0253-4781-9ffd-3e7098d6068b-kube-api-access-mfjdl" (OuterVolumeSpecName: "kube-api-access-mfjdl") pod "1bbddd27-0253-4781-9ffd-3e7098d6068b" (UID: "1bbddd27-0253-4781-9ffd-3e7098d6068b"). InnerVolumeSpecName "kube-api-access-mfjdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:11:25.313716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.313697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:25.321944 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.321919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:25.407332 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.407298    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1bbddd27-0253-4781-9ffd-3e7098d6068b-extensions-socket-volume\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:25.407332 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.407329    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfjdl\" (UniqueName: \"kubernetes.io/projected/1bbddd27-0253-4781-9ffd-3e7098d6068b-kube-api-access-mfjdl\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:25.461560 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.461537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"]
Jul 08 19:11:25.465600 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:11:25.465563    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0aece5_ac8e_4d1d_99e0_2f6d47b5c8b4.slice/crio-a90eb6ffa63237d39a6c582a37202f881920103aea73533400766189dd2cf1e0 WatchSource:0}: Error finding container a90eb6ffa63237d39a6c582a37202f881920103aea73533400766189dd2cf1e0: Status 404 returned error can't find the container with id a90eb6ffa63237d39a6c582a37202f881920103aea73533400766189dd2cf1e0
Jul 08 19:11:25.480589 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:25.480568    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"]
Jul 08 19:11:25.482933 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:11:25.482898    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc996b6b9_b03d_422d_9c22_713736a39756.slice/crio-b893ec544cf4bbecf5c91727c27d0b7977b9c7e05605c7b7898fa49bfb240b85 WatchSource:0}: Error finding container b893ec544cf4bbecf5c91727c27d0b7977b9c7e05605c7b7898fa49bfb240b85: Status 404 returned error can't find the container with id b893ec544cf4bbecf5c91727c27d0b7977b9c7e05605c7b7898fa49bfb240b85
Jul 08 19:11:25.533710 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.533675    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:25.535987 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:25.535962    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:26.201688 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.201656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n" event={"ID":"c996b6b9-b03d-422d-9c22-713736a39756","Type":"ContainerStarted","Data":"2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162"}
Jul 08 19:11:26.202098 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.201715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n" event={"ID":"c996b6b9-b03d-422d-9c22-713736a39756","Type":"ContainerStarted","Data":"b893ec544cf4bbecf5c91727c27d0b7977b9c7e05605c7b7898fa49bfb240b85"}
Jul 08 19:11:26.202098 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.201833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:26.203197 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.203173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j" event={"ID":"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4","Type":"ContainerStarted","Data":"cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e"}
Jul 08 19:11:26.203300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.203213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j" event={"ID":"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4","Type":"ContainerStarted","Data":"a90eb6ffa63237d39a6c582a37202f881920103aea73533400766189dd2cf1e0"}
Jul 08 19:11:26.203300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.203295    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:26.204276 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:26.204255    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:26.204784 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.204760    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" containerName="manager" containerID="cri-o://5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b" gracePeriod=2
Jul 08 19:11:26.206593 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:26.206561    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:26.233809 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.233769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n" podStartSLOduration=2.233755813 podStartE2EDuration="2.233755813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:26.231892551 +0000 UTC m=+744.875913281" watchObservedRunningTime="2026-07-08 19:11:26.233755813 +0000 UTC m=+744.877776544"
Jul 08 19:11:26.234135 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:26.234107    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:26.257372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.257323    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j" podStartSLOduration=2.25729867 podStartE2EDuration="2.25729867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:26.255821158 +0000 UTC m=+744.899841890" watchObservedRunningTime="2026-07-08 19:11:26.25729867 +0000 UTC m=+744.901319401"
Jul 08 19:11:26.258102 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:26.258073    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:26.456170 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.455803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:26.458874 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:26.458836    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:26.459951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.459932    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" path="/var/lib/kubelet/pods/1bbddd27-0253-4781-9ffd-3e7098d6068b/volumes"
Jul 08 19:11:26.461724 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:26.461702    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jp692\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="1bbddd27-0253-4781-9ffd-3e7098d6068b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jp692"
Jul 08 19:11:26.517178 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.517154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sxfq\" (UniqueName: \"kubernetes.io/projected/d98aeb19-35b1-4900-af99-1b05cb3450c4-kube-api-access-5sxfq\") pod \"d98aeb19-35b1-4900-af99-1b05cb3450c4\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") "
Jul 08 19:11:26.517285 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.517258    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d98aeb19-35b1-4900-af99-1b05cb3450c4-extensions-socket-volume\") pod \"d98aeb19-35b1-4900-af99-1b05cb3450c4\" (UID: \"d98aeb19-35b1-4900-af99-1b05cb3450c4\") "
Jul 08 19:11:26.517711 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.517652    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98aeb19-35b1-4900-af99-1b05cb3450c4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d98aeb19-35b1-4900-af99-1b05cb3450c4" (UID: "d98aeb19-35b1-4900-af99-1b05cb3450c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:11:26.519253 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.519233    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98aeb19-35b1-4900-af99-1b05cb3450c4-kube-api-access-5sxfq" (OuterVolumeSpecName: "kube-api-access-5sxfq") pod "d98aeb19-35b1-4900-af99-1b05cb3450c4" (UID: "d98aeb19-35b1-4900-af99-1b05cb3450c4"). InnerVolumeSpecName "kube-api-access-5sxfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:11:26.618745 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.618714    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d98aeb19-35b1-4900-af99-1b05cb3450c4-extensions-socket-volume\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:26.618745 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.618741    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxfq\" (UniqueName: \"kubernetes.io/projected/d98aeb19-35b1-4900-af99-1b05cb3450c4-kube-api-access-5sxfq\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:26.942026 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:26.941991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:27.046568 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:27.046537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:27.211065 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:27.210983    2563 generic.go:356] "Generic (PLEG): container finished" podID="d98aeb19-35b1-4900-af99-1b05cb3450c4" containerID="5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b" exitCode=0
Jul 08 19:11:27.211065 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:27.211028    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:27.211504 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:27.211071    2563 scope.go:117] "RemoveContainer" containerID="5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b"
Jul 08 19:11:27.220217 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:27.220202    2563 scope.go:117] "RemoveContainer" containerID="5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b"
Jul 08 19:11:27.220459 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:27.220442    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b\": container with ID starting with 5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b not found: ID does not exist" containerID="5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b"
Jul 08 19:11:27.220510 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:27.220467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b"} err="failed to get container status \"5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b\": rpc error: code = NotFound desc = could not find container \"5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b\": container with ID starting with 5ceb533bbce4590628ead5ff737b6a0df04a629b2bff7641ff6acaa01027a99b not found: ID does not exist"
Jul 08 19:11:27.222084 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:27.222064    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sh52k\" is forbidden: User \"system:node:ip-10-0-128-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-89.ec2.internal' and this object" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sh52k"
Jul 08 19:11:28.454890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:28.454851    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98aeb19-35b1-4900-af99-1b05cb3450c4" path="/var/lib/kubelet/pods/d98aeb19-35b1-4900-af99-1b05cb3450c4/volumes"
Jul 08 19:11:29.778612 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:29.778580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:29.880156 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:29.880126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:33.571197 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:33.571162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:33.630198 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:33.630167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:36.455979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:36.455944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:37.213971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.213934    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:37.217684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.214678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:37.283736 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.283700    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"]
Jul 08 19:11:37.284030 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.284003    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n" podUID="c996b6b9-b03d-422d-9c22-713736a39756" containerName="manager" containerID="cri-o://2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162" gracePeriod=10
Jul 08 19:11:37.521719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.521690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:37.612428 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.612399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c996b6b9-b03d-422d-9c22-713736a39756-extensions-socket-volume\") pod \"c996b6b9-b03d-422d-9c22-713736a39756\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") "
Jul 08 19:11:37.612575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.612454    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nhtq\" (UniqueName: \"kubernetes.io/projected/c996b6b9-b03d-422d-9c22-713736a39756-kube-api-access-6nhtq\") pod \"c996b6b9-b03d-422d-9c22-713736a39756\" (UID: \"c996b6b9-b03d-422d-9c22-713736a39756\") "
Jul 08 19:11:37.612823 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.612791    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c996b6b9-b03d-422d-9c22-713736a39756-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c996b6b9-b03d-422d-9c22-713736a39756" (UID: "c996b6b9-b03d-422d-9c22-713736a39756"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:11:37.614316 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.614286    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c996b6b9-b03d-422d-9c22-713736a39756-kube-api-access-6nhtq" (OuterVolumeSpecName: "kube-api-access-6nhtq") pod "c996b6b9-b03d-422d-9c22-713736a39756" (UID: "c996b6b9-b03d-422d-9c22-713736a39756"). InnerVolumeSpecName "kube-api-access-6nhtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:11:37.713558 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.713519    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c996b6b9-b03d-422d-9c22-713736a39756-extensions-socket-volume\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:37.713558 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:37.713553    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhtq\" (UniqueName: \"kubernetes.io/projected/c996b6b9-b03d-422d-9c22-713736a39756-kube-api-access-6nhtq\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:38.269737 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.269704    2563 generic.go:356] "Generic (PLEG): container finished" podID="c996b6b9-b03d-422d-9c22-713736a39756" containerID="2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162" exitCode=0
Jul 08 19:11:38.269905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.269746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"
Jul 08 19:11:38.269905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.269764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n" event={"ID":"c996b6b9-b03d-422d-9c22-713736a39756","Type":"ContainerDied","Data":"2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162"}
Jul 08 19:11:38.269905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.269808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n" event={"ID":"c996b6b9-b03d-422d-9c22-713736a39756","Type":"ContainerDied","Data":"b893ec544cf4bbecf5c91727c27d0b7977b9c7e05605c7b7898fa49bfb240b85"}
Jul 08 19:11:38.269905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.269829    2563 scope.go:117] "RemoveContainer" containerID="2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162"
Jul 08 19:11:38.279467 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.279447    2563 scope.go:117] "RemoveContainer" containerID="2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162"
Jul 08 19:11:38.279731 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:38.279713    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162\": container with ID starting with 2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162 not found: ID does not exist" containerID="2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162"
Jul 08 19:11:38.279793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.279740    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162"} err="failed to get container status \"2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162\": rpc error: code = NotFound desc = could not find container \"2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162\": container with ID starting with 2eee9ae19aa44663b975780ffdb2e361070f4bab3d78ba7b5724d587b4841162 not found: ID does not exist"
Jul 08 19:11:38.297421 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.297397    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"]
Jul 08 19:11:38.304287 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.304264    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cc2n"]
Jul 08 19:11:38.455201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:38.455167    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c996b6b9-b03d-422d-9c22-713736a39756" path="/var/lib/kubelet/pods/c996b6b9-b03d-422d-9c22-713736a39756/volumes"
Jul 08 19:11:42.774659 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:42.774609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:42.875840 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:42.875804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:53.651533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.651499    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"]
Jul 08 19:11:53.651992 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.651959    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c996b6b9-b03d-422d-9c22-713736a39756" containerName="manager"
Jul 08 19:11:53.651992 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.651974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c996b6b9-b03d-422d-9c22-713736a39756" containerName="manager"
Jul 08 19:11:53.652116 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.652066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c996b6b9-b03d-422d-9c22-713736a39756" containerName="manager"
Jul 08 19:11:53.654910 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.654889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.657762 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.657740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p222s\""
Jul 08 19:11:53.668366 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.668338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"]
Jul 08 19:11:53.682610 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.682576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:53.743123 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5k4\" (UniqueName: \"kubernetes.io/projected/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-kube-api-access-9c5k4\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743454 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743454 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743454 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.743454 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.743379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.843790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.843737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.843790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.843785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.843790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.843807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5k4\" (UniqueName: \"kubernetes.io/projected/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-kube-api-access-9c5k4\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844102 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.843834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844102 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.843882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844102 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.843910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844102 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.843968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844102 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.844015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844102 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.844052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844363 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.844221    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844363 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.844283    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844480 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.844460    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844517 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.844490    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.844603 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.844573    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.846460 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.846430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.846538 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.846503    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.852790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.852737    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.853152 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.853134    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5k4\" (UniqueName: \"kubernetes.io/projected/8dd6eb0d-e25c-4fb5-9a14-c85c50626e61-kube-api-access-9c5k4\") pod \"maas-default-gateway-openshift-default-56654dcf7-cdrm2\" (UID: \"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:53.964691 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:53.964580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:54.106608 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.106585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"]
Jul 08 19:11:54.108586 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:11:54.108542    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd6eb0d_e25c_4fb5_9a14_c85c50626e61.slice/crio-c8bb982aba34d70ed72c7bb90212a06059347a7b8ee865d267fae20ebf0f327d WatchSource:0}: Error finding container c8bb982aba34d70ed72c7bb90212a06059347a7b8ee865d267fae20ebf0f327d: Status 404 returned error can't find the container with id c8bb982aba34d70ed72c7bb90212a06059347a7b8ee865d267fae20ebf0f327d
Jul 08 19:11:54.110768 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.110736    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:11:54.110870 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.110813    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:11:54.110870 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.110856    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:11:54.176543 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.176511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:54.349466 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.349429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" event={"ID":"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61","Type":"ContainerStarted","Data":"f98b2c7e537fc36ca16ac416d89420a8e1f09da192af091e4ca0f2d5fd97c49f"}
Jul 08 19:11:54.349672 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.349480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" event={"ID":"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61","Type":"ContainerStarted","Data":"c8bb982aba34d70ed72c7bb90212a06059347a7b8ee865d267fae20ebf0f327d"}
Jul 08 19:11:54.371730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.371685    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" podStartSLOduration=1.3716702490000001 podStartE2EDuration="1.371670249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:54.370111219 +0000 UTC m=+773.014131951" watchObservedRunningTime="2026-07-08 19:11:54.371670249 +0000 UTC m=+773.015690979"
Jul 08 19:11:54.964990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:54.964956    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:55.352120 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.352089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:55.363387 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.363359    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pf952"]
Jul 08 19:11:55.365781 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.365762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.368599 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.368573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lhrps\""
Jul 08 19:11:55.368709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.368602    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 08 19:11:55.375468 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.375446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pf952"]
Jul 08 19:11:55.456960 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.456927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pf952"]
Jul 08 19:11:55.459347 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.459314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-config-file\") pod \"limitador-limitador-64c94fd64f-pf952\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.459509 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.459364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7n6m\" (UniqueName: \"kubernetes.io/projected/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-kube-api-access-b7n6m\") pod \"limitador-limitador-64c94fd64f-pf952\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.464752 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.464722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:55.561099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.561004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-config-file\") pod \"limitador-limitador-64c94fd64f-pf952\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.561099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.561061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7n6m\" (UniqueName: \"kubernetes.io/projected/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-kube-api-access-b7n6m\") pod \"limitador-limitador-64c94fd64f-pf952\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.562162 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.562132    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-config-file\") pod \"limitador-limitador-64c94fd64f-pf952\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.565951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.565927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:55.569760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.569740    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7n6m\" (UniqueName: \"kubernetes.io/projected/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-kube-api-access-b7n6m\") pod \"limitador-limitador-64c94fd64f-pf952\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.677521 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.677430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:55.809913 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.809890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pf952"]
Jul 08 19:11:55.811817 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:11:55.811791    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b96b1a5_8af3_40b4_ac36_38515adb0ab7.slice/crio-b591cacf8499e3b46bab9ae69a6d665eb671d11332c0625f09fa0a136ef6da0f WatchSource:0}: Error finding container b591cacf8499e3b46bab9ae69a6d665eb671d11332c0625f09fa0a136ef6da0f: Status 404 returned error can't find the container with id b591cacf8499e3b46bab9ae69a6d665eb671d11332c0625f09fa0a136ef6da0f
Jul 08 19:11:55.970238 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:55.970151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:56.186719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.186685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:56.250971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.250941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:11:56.362048 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.362015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952" event={"ID":"6b96b1a5-8af3-40b4-ac36-38515adb0ab7","Type":"ContainerStarted","Data":"b591cacf8499e3b46bab9ae69a6d665eb671d11332c0625f09fa0a136ef6da0f"}
Jul 08 19:11:56.362233 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.362217    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:56.363300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.363281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:11:56.364234 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.364208    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"]
Jul 08 19:11:56.364384 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.364364    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j" podUID="ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" containerName="manager" containerID="cri-o://cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e" gracePeriod=10
Jul 08 19:11:56.657097 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.656909    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:56.671429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.671394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-extensions-socket-volume\") pod \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") "
Jul 08 19:11:56.671582 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.671522    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lt2k\" (UniqueName: \"kubernetes.io/projected/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-kube-api-access-6lt2k\") pod \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\" (UID: \"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4\") "
Jul 08 19:11:56.671869 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.671838    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" (UID: "ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:11:56.674958 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.674919    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-kube-api-access-6lt2k" (OuterVolumeSpecName: "kube-api-access-6lt2k") pod "ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" (UID: "ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4"). InnerVolumeSpecName "kube-api-access-6lt2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:11:56.773078 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.773026    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-extensions-socket-volume\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:56.773078 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:56.773058    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lt2k\" (UniqueName: \"kubernetes.io/projected/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4-kube-api-access-6lt2k\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:11:57.369871 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:57.369834    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" containerID="cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e" exitCode=0
Jul 08 19:11:57.370332 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:57.369889    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"
Jul 08 19:11:57.370332 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:57.369922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j" event={"ID":"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4","Type":"ContainerDied","Data":"cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e"}
Jul 08 19:11:57.370332 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:57.369973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j" event={"ID":"ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4","Type":"ContainerDied","Data":"a90eb6ffa63237d39a6c582a37202f881920103aea73533400766189dd2cf1e0"}
Jul 08 19:11:57.370332 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:57.370002    2563 scope.go:117] "RemoveContainer" containerID="cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e"
Jul 08 19:11:57.396820 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:57.396786    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"]
Jul 08 19:11:57.401008 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:57.400980    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qh8j"]
Jul 08 19:11:58.019758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:58.018012    2563 scope.go:117] "RemoveContainer" containerID="cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e"
Jul 08 19:11:58.019758 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:11:58.018439    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e\": container with ID starting with cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e not found: ID does not exist" containerID="cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e"
Jul 08 19:11:58.019758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:58.018475    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e"} err="failed to get container status \"cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e\": rpc error: code = NotFound desc = could not find container \"cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e\": container with ID starting with cfd480a6fab1065639133ef92becd0b8391d517d80ee1134ee2256d1cde2436e not found: ID does not exist"
Jul 08 19:11:58.456084 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:58.456013    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" path="/var/lib/kubelet/pods/ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4/volumes"
Jul 08 19:11:59.385752 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:59.385717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952" event={"ID":"6b96b1a5-8af3-40b4-ac36-38515adb0ab7","Type":"ContainerStarted","Data":"af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d"}
Jul 08 19:11:59.386214 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:59.385858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:11:59.402989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:11:59.402937    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952" podStartSLOduration=1.653480435 podStartE2EDuration="4.402918173s" totalImagesPullingTime="2.749437738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:59.402620035 +0000 UTC m=+778.046640767" watchObservedRunningTime="2026-07-08 19:11:59.402918173 +0000 UTC m=+778.046938905"
Jul 08 19:12:04.697359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:04.697325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:10.207311 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.207278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pf952"]
Jul 08 19:12:10.207724 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.207473    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952" podUID="6b96b1a5-8af3-40b4-ac36-38515adb0ab7" containerName="limitador" containerID="cri-o://af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d" gracePeriod=30
Jul 08 19:12:10.208229 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.208184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:12:10.743825 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.743796    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:12:10.791997 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.791929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7n6m\" (UniqueName: \"kubernetes.io/projected/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-kube-api-access-b7n6m\") pod \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") "
Jul 08 19:12:10.791997 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.791965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-config-file\") pod \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\" (UID: \"6b96b1a5-8af3-40b4-ac36-38515adb0ab7\") "
Jul 08 19:12:10.792387 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.792354    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-config-file" (OuterVolumeSpecName: "config-file") pod "6b96b1a5-8af3-40b4-ac36-38515adb0ab7" (UID: "6b96b1a5-8af3-40b4-ac36-38515adb0ab7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:12:10.794027 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.794004    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-kube-api-access-b7n6m" (OuterVolumeSpecName: "kube-api-access-b7n6m") pod "6b96b1a5-8af3-40b4-ac36-38515adb0ab7" (UID: "6b96b1a5-8af3-40b4-ac36-38515adb0ab7"). InnerVolumeSpecName "kube-api-access-b7n6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:12:10.893331 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.893294    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7n6m\" (UniqueName: \"kubernetes.io/projected/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-kube-api-access-b7n6m\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:12:10.893331 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:10.893325    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6b96b1a5-8af3-40b4-ac36-38515adb0ab7-config-file\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:12:11.444640 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.444584    2563 generic.go:356] "Generic (PLEG): container finished" podID="6b96b1a5-8af3-40b4-ac36-38515adb0ab7" containerID="af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d" exitCode=0
Jul 08 19:12:11.445021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.444655    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952"
Jul 08 19:12:11.445021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.444682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952" event={"ID":"6b96b1a5-8af3-40b4-ac36-38515adb0ab7","Type":"ContainerDied","Data":"af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d"}
Jul 08 19:12:11.445021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.444749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pf952" event={"ID":"6b96b1a5-8af3-40b4-ac36-38515adb0ab7","Type":"ContainerDied","Data":"b591cacf8499e3b46bab9ae69a6d665eb671d11332c0625f09fa0a136ef6da0f"}
Jul 08 19:12:11.445021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.444779    2563 scope.go:117] "RemoveContainer" containerID="af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d"
Jul 08 19:12:11.453874 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.453855    2563 scope.go:117] "RemoveContainer" containerID="af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d"
Jul 08 19:12:11.454175 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:12:11.454156    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d\": container with ID starting with af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d not found: ID does not exist" containerID="af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d"
Jul 08 19:12:11.454223 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.454187    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d"} err="failed to get container status \"af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d\": rpc error: code = NotFound desc = could not find container \"af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d\": container with ID starting with af657dca3749b622023c2c2007ef1e92f64725e64052a7840432b240dc442f6d not found: ID does not exist"
Jul 08 19:12:11.468018 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.467992    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pf952"]
Jul 08 19:12:11.469809 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:11.469786    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pf952"]
Jul 08 19:12:12.454973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:12.454942    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b96b1a5-8af3-40b4-ac36-38515adb0ab7" path="/var/lib/kubelet/pods/6b96b1a5-8af3-40b4-ac36-38515adb0ab7/volumes"
Jul 08 19:12:17.799326 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.799291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-qzsml"]
Jul 08 19:12:17.799719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.799577    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" containerName="manager"
Jul 08 19:12:17.799719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.799588    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" containerName="manager"
Jul 08 19:12:17.799719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.799595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b96b1a5-8af3-40b4-ac36-38515adb0ab7" containerName="limitador"
Jul 08 19:12:17.799719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.799601    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b96b1a5-8af3-40b4-ac36-38515adb0ab7" containerName="limitador"
Jul 08 19:12:17.799719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.799681    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b96b1a5-8af3-40b4-ac36-38515adb0ab7" containerName="limitador"
Jul 08 19:12:17.799719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.799692    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0aece5-ac8e-4d1d-99e0-2f6d47b5c8b4" containerName="manager"
Jul 08 19:12:17.803908 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.803890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qzsml"
Jul 08 19:12:17.807283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.807262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r57js\""
Jul 08 19:12:17.810979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.810741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qzsml"]
Jul 08 19:12:17.953763 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:17.953731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296wd\" (UniqueName: \"kubernetes.io/projected/b7bafff0-7a64-4ba1-a0d5-e587afb56a39-kube-api-access-296wd\") pod \"authorino-75d556bd77-qzsml\" (UID: \"b7bafff0-7a64-4ba1-a0d5-e587afb56a39\") " pod="kuadrant-system/authorino-75d556bd77-qzsml"
Jul 08 19:12:18.054299 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:18.054212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296wd\" (UniqueName: \"kubernetes.io/projected/b7bafff0-7a64-4ba1-a0d5-e587afb56a39-kube-api-access-296wd\") pod \"authorino-75d556bd77-qzsml\" (UID: \"b7bafff0-7a64-4ba1-a0d5-e587afb56a39\") " pod="kuadrant-system/authorino-75d556bd77-qzsml"
Jul 08 19:12:18.062418 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:18.062386    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296wd\" (UniqueName: \"kubernetes.io/projected/b7bafff0-7a64-4ba1-a0d5-e587afb56a39-kube-api-access-296wd\") pod \"authorino-75d556bd77-qzsml\" (UID: \"b7bafff0-7a64-4ba1-a0d5-e587afb56a39\") " pod="kuadrant-system/authorino-75d556bd77-qzsml"
Jul 08 19:12:18.113360 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:18.113334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qzsml"
Jul 08 19:12:18.240057 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:18.240019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qzsml"]
Jul 08 19:12:18.241109 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:12:18.241085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7bafff0_7a64_4ba1_a0d5_e587afb56a39.slice/crio-ba40e0b31864e1073f40a9cbc5927954beea46cd87b3408eb11ffc4dfa82dbc7 WatchSource:0}: Error finding container ba40e0b31864e1073f40a9cbc5927954beea46cd87b3408eb11ffc4dfa82dbc7: Status 404 returned error can't find the container with id ba40e0b31864e1073f40a9cbc5927954beea46cd87b3408eb11ffc4dfa82dbc7
Jul 08 19:12:18.478303 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:18.478221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qzsml" event={"ID":"b7bafff0-7a64-4ba1-a0d5-e587afb56a39","Type":"ContainerStarted","Data":"ba40e0b31864e1073f40a9cbc5927954beea46cd87b3408eb11ffc4dfa82dbc7"}
Jul 08 19:12:21.750389 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:21.750357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:21.845096 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:21.845045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:22.161078 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.161046    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"]
Jul 08 19:12:22.164471 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.164455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.167814 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.167781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-hqdjm\""
Jul 08 19:12:22.167814 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.167782    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 08 19:12:22.167976 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.167851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 08 19:12:22.168012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.168003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 08 19:12:22.172896 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.172872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"]
Jul 08 19:12:22.186586 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.186534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:22.292536 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.292499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6cl\" (UniqueName: \"kubernetes.io/projected/58ac526b-e757-4cd5-a56d-727fcb1a18a8-kube-api-access-cq6cl\") pod \"postgres-69fb4594cf-rnqsb\" (UID: \"58ac526b-e757-4cd5-a56d-727fcb1a18a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.292712 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.292580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/58ac526b-e757-4cd5-a56d-727fcb1a18a8-data\") pod \"postgres-69fb4594cf-rnqsb\" (UID: \"58ac526b-e757-4cd5-a56d-727fcb1a18a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.393696 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.393665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/58ac526b-e757-4cd5-a56d-727fcb1a18a8-data\") pod \"postgres-69fb4594cf-rnqsb\" (UID: \"58ac526b-e757-4cd5-a56d-727fcb1a18a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.393854 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.393723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6cl\" (UniqueName: \"kubernetes.io/projected/58ac526b-e757-4cd5-a56d-727fcb1a18a8-kube-api-access-cq6cl\") pod \"postgres-69fb4594cf-rnqsb\" (UID: \"58ac526b-e757-4cd5-a56d-727fcb1a18a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.394040 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.394020    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/58ac526b-e757-4cd5-a56d-727fcb1a18a8-data\") pod \"postgres-69fb4594cf-rnqsb\" (UID: \"58ac526b-e757-4cd5-a56d-727fcb1a18a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.403492 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.403467    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6cl\" (UniqueName: \"kubernetes.io/projected/58ac526b-e757-4cd5-a56d-727fcb1a18a8-kube-api-access-cq6cl\") pod \"postgres-69fb4594cf-rnqsb\" (UID: \"58ac526b-e757-4cd5-a56d-727fcb1a18a8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.474876 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.474806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:22.501991 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.501958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qzsml" event={"ID":"b7bafff0-7a64-4ba1-a0d5-e587afb56a39","Type":"ContainerStarted","Data":"57e1f44c75810b53a7fdd07ed74b0d5c6e27480eec48190e269964eba856b630"}
Jul 08 19:12:22.519602 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.519558    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-qzsml" podStartSLOduration=2.052808167 podStartE2EDuration="5.519539304s" totalImagesPullingTime="3.466731137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:22.518208029 +0000 UTC m=+801.162228760" watchObservedRunningTime="2026-07-08 19:12:22.519539304 +0000 UTC m=+801.163560036"
Jul 08 19:12:22.612335 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:22.612311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"]
Jul 08 19:12:22.613784 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:12:22.613750    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ac526b_e757_4cd5_a56d_727fcb1a18a8.slice/crio-2136d6f33cae9220ebc64d3211599408090fca14e10cbd38a36cc5d13b61bb4c WatchSource:0}: Error finding container 2136d6f33cae9220ebc64d3211599408090fca14e10cbd38a36cc5d13b61bb4c: Status 404 returned error can't find the container with id 2136d6f33cae9220ebc64d3211599408090fca14e10cbd38a36cc5d13b61bb4c
Jul 08 19:12:23.507906 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:23.507821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb" event={"ID":"58ac526b-e757-4cd5-a56d-727fcb1a18a8","Type":"ContainerStarted","Data":"2136d6f33cae9220ebc64d3211599408090fca14e10cbd38a36cc5d13b61bb4c"}
Jul 08 19:12:28.542270 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:28.542233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb" event={"ID":"58ac526b-e757-4cd5-a56d-727fcb1a18a8","Type":"ContainerStarted","Data":"37c39fcb4451ceed98adb1d9e305b9a153e26666574423cdb13cf9d39e0961fe"}
Jul 08 19:12:28.542702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:28.542371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:28.563083 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:28.563022    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb" podStartSLOduration=1.457617142 podStartE2EDuration="6.563003571s" totalImagesPullingTime="5.105386429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:28.559671282 +0000 UTC m=+807.203692009" watchObservedRunningTime="2026-07-08 19:12:28.563003571 +0000 UTC m=+807.207024309"
Jul 08 19:12:34.578840 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:34.578812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rnqsb"
Jul 08 19:12:34.893652 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:34.893550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:35.238336 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:35.238242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:37.366533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.366501    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qzsml"]
Jul 08 19:12:37.366926 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.366694    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-qzsml" podUID="b7bafff0-7a64-4ba1-a0d5-e587afb56a39" containerName="authorino" containerID="cri-o://57e1f44c75810b53a7fdd07ed74b0d5c6e27480eec48190e269964eba856b630" gracePeriod=30
Jul 08 19:12:37.591700 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.591665    2563 generic.go:356] "Generic (PLEG): container finished" podID="b7bafff0-7a64-4ba1-a0d5-e587afb56a39" containerID="57e1f44c75810b53a7fdd07ed74b0d5c6e27480eec48190e269964eba856b630" exitCode=0
Jul 08 19:12:37.591860 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.591747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qzsml" event={"ID":"b7bafff0-7a64-4ba1-a0d5-e587afb56a39","Type":"ContainerDied","Data":"57e1f44c75810b53a7fdd07ed74b0d5c6e27480eec48190e269964eba856b630"}
Jul 08 19:12:37.613210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.613187    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qzsml"
Jul 08 19:12:37.728008 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.727908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-296wd\" (UniqueName: \"kubernetes.io/projected/b7bafff0-7a64-4ba1-a0d5-e587afb56a39-kube-api-access-296wd\") pod \"b7bafff0-7a64-4ba1-a0d5-e587afb56a39\" (UID: \"b7bafff0-7a64-4ba1-a0d5-e587afb56a39\") "
Jul 08 19:12:37.729978 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.729952    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bafff0-7a64-4ba1-a0d5-e587afb56a39-kube-api-access-296wd" (OuterVolumeSpecName: "kube-api-access-296wd") pod "b7bafff0-7a64-4ba1-a0d5-e587afb56a39" (UID: "b7bafff0-7a64-4ba1-a0d5-e587afb56a39"). InnerVolumeSpecName "kube-api-access-296wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:12:37.828874 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:37.828835    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296wd\" (UniqueName: \"kubernetes.io/projected/b7bafff0-7a64-4ba1-a0d5-e587afb56a39-kube-api-access-296wd\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:12:38.597634 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:38.597604    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-qzsml"
Jul 08 19:12:38.598070 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:38.597648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-qzsml" event={"ID":"b7bafff0-7a64-4ba1-a0d5-e587afb56a39","Type":"ContainerDied","Data":"ba40e0b31864e1073f40a9cbc5927954beea46cd87b3408eb11ffc4dfa82dbc7"}
Jul 08 19:12:38.598070 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:38.597698    2563 scope.go:117] "RemoveContainer" containerID="57e1f44c75810b53a7fdd07ed74b0d5c6e27480eec48190e269964eba856b630"
Jul 08 19:12:38.615613 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:38.615590    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qzsml"]
Jul 08 19:12:38.619514 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:38.619496    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-qzsml"]
Jul 08 19:12:40.454720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:40.454680    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7bafff0-7a64-4ba1-a0d5-e587afb56a39" path="/var/lib/kubelet/pods/b7bafff0-7a64-4ba1-a0d5-e587afb56a39/volumes"
Jul 08 19:12:41.761094 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.761065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:41.792734 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.792701    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-746cdc56fc-xpvd9"]
Jul 08 19:12:41.793031 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.793018    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7bafff0-7a64-4ba1-a0d5-e587afb56a39" containerName="authorino"
Jul 08 19:12:41.793076 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.793033    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7bafff0-7a64-4ba1-a0d5-e587afb56a39" containerName="authorino"
Jul 08 19:12:41.793110 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.793103    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7bafff0-7a64-4ba1-a0d5-e587afb56a39" containerName="authorino"
Jul 08 19:12:41.795893 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.795877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:41.799046 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.799022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 08 19:12:41.799172 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.799029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vkzct\""
Jul 08 19:12:41.799228 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.799193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 08 19:12:41.807234 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.807212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-746cdc56fc-xpvd9"]
Jul 08 19:12:41.861029 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.860993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert\") pod \"maas-controller-746cdc56fc-xpvd9\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") " pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:41.861192 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.861038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-592qr\" (UniqueName: \"kubernetes.io/projected/d6b9e911-6989-44f5-a261-79334324488c-kube-api-access-592qr\") pod \"maas-controller-746cdc56fc-xpvd9\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") " pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:41.886801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.886772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:41.962276 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.962242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert\") pod \"maas-controller-746cdc56fc-xpvd9\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") " pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:41.962276 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.962282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-592qr\" (UniqueName: \"kubernetes.io/projected/d6b9e911-6989-44f5-a261-79334324488c-kube-api-access-592qr\") pod \"maas-controller-746cdc56fc-xpvd9\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") " pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:41.962557 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:12:41.962394    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 08 19:12:41.962557 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:12:41.962471    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert podName:d6b9e911-6989-44f5-a261-79334324488c nodeName:}" failed. No retries permitted until 2026-07-08 19:12:42.462450582 +0000 UTC m=+821.106471294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert") pod "maas-controller-746cdc56fc-xpvd9" (UID: "d6b9e911-6989-44f5-a261-79334324488c") : secret "maas-controller-webhook-cert" not found
Jul 08 19:12:41.974088 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.974055    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-592qr\" (UniqueName: \"kubernetes.io/projected/d6b9e911-6989-44f5-a261-79334324488c-kube-api-access-592qr\") pod \"maas-controller-746cdc56fc-xpvd9\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") " pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:41.974927 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.974907    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-7ngcr"]
Jul 08 19:12:41.977137 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.977118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:41.989848 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.989819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-7ngcr"]
Jul 08 19:12:41.993816 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:41.993793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:42.063320 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.063205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24ab442b-1fce-409a-bc8e-51c3ca8707e3-webhook-cert\") pod \"maas-controller-7fcfb68c87-7ngcr\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") " pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:42.063488 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.063327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jt9\" (UniqueName: \"kubernetes.io/projected/24ab442b-1fce-409a-bc8e-51c3ca8707e3-kube-api-access-s9jt9\") pod \"maas-controller-7fcfb68c87-7ngcr\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") " pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:42.090875 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.090838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:42.164342 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.164308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jt9\" (UniqueName: \"kubernetes.io/projected/24ab442b-1fce-409a-bc8e-51c3ca8707e3-kube-api-access-s9jt9\") pod \"maas-controller-7fcfb68c87-7ngcr\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") " pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:42.164544 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.164420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24ab442b-1fce-409a-bc8e-51c3ca8707e3-webhook-cert\") pod \"maas-controller-7fcfb68c87-7ngcr\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") " pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:42.166939 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.166917    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24ab442b-1fce-409a-bc8e-51c3ca8707e3-webhook-cert\") pod \"maas-controller-7fcfb68c87-7ngcr\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") " pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:42.173726 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.173696    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jt9\" (UniqueName: \"kubernetes.io/projected/24ab442b-1fce-409a-bc8e-51c3ca8707e3-kube-api-access-s9jt9\") pod \"maas-controller-7fcfb68c87-7ngcr\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") " pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:42.289570 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.289530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:42.422462 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.422437    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-7ngcr"]
Jul 08 19:12:42.424072 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:12:42.424044    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ab442b_1fce_409a_bc8e_51c3ca8707e3.slice/crio-cf26fa02c1768c91532165384828ba346ec8c2ade4aa2b2b2a6dd8a15de3d298 WatchSource:0}: Error finding container cf26fa02c1768c91532165384828ba346ec8c2ade4aa2b2b2a6dd8a15de3d298: Status 404 returned error can't find the container with id cf26fa02c1768c91532165384828ba346ec8c2ade4aa2b2b2a6dd8a15de3d298
Jul 08 19:12:42.467077 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.467045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert\") pod \"maas-controller-746cdc56fc-xpvd9\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") " pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:42.469408 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.469380    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert\") pod \"maas-controller-746cdc56fc-xpvd9\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") " pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:42.623669 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.623563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr" event={"ID":"24ab442b-1fce-409a-bc8e-51c3ca8707e3","Type":"ContainerStarted","Data":"cf26fa02c1768c91532165384828ba346ec8c2ade4aa2b2b2a6dd8a15de3d298"}
Jul 08 19:12:42.708924 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.708888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:42.840200 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:42.840171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-746cdc56fc-xpvd9"]
Jul 08 19:12:42.841561 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:12:42.841531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b9e911_6989_44f5_a261_79334324488c.slice/crio-ec9e06ff6d1684a0b7d8e0d71eb9103259d9b958016844ed0ff90d947b7912d2 WatchSource:0}: Error finding container ec9e06ff6d1684a0b7d8e0d71eb9103259d9b958016844ed0ff90d947b7912d2: Status 404 returned error can't find the container with id ec9e06ff6d1684a0b7d8e0d71eb9103259d9b958016844ed0ff90d947b7912d2
Jul 08 19:12:43.632787 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:43.632745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-746cdc56fc-xpvd9" event={"ID":"d6b9e911-6989-44f5-a261-79334324488c","Type":"ContainerStarted","Data":"ec9e06ff6d1684a0b7d8e0d71eb9103259d9b958016844ed0ff90d947b7912d2"}
Jul 08 19:12:44.969702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:44.969664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:45.053238 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:45.053187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:12:45.643886 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:45.643850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr" event={"ID":"24ab442b-1fce-409a-bc8e-51c3ca8707e3","Type":"ContainerStarted","Data":"4b56904e06d80e3916bd97851367b4434b466ff08531e234a791c9bb72fd672f"}
Jul 08 19:12:45.644101 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:45.643974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:45.645456 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:45.645423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-746cdc56fc-xpvd9" event={"ID":"d6b9e911-6989-44f5-a261-79334324488c","Type":"ContainerStarted","Data":"48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682"}
Jul 08 19:12:45.645549 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:45.645512    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:45.663221 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:45.663177    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr" podStartSLOduration=2.345770312 podStartE2EDuration="4.663163271s" totalImagesPullingTime="2.317392959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:45.660917319 +0000 UTC m=+824.304938051" watchObservedRunningTime="2026-07-08 19:12:45.663163271 +0000 UTC m=+824.307183979"
Jul 08 19:12:45.679552 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:45.679505    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-746cdc56fc-xpvd9" podStartSLOduration=2.7698181 podStartE2EDuration="4.679491889s" totalImagesPullingTime="1.909673789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:45.678277171 +0000 UTC m=+824.322297901" watchObservedRunningTime="2026-07-08 19:12:45.679491889 +0000 UTC m=+824.323512624"
Jul 08 19:12:56.655887 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:56.655854    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:56.656272 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:56.656133    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:12:56.724632 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:56.724600    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-746cdc56fc-xpvd9"]
Jul 08 19:12:56.724899 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:56.724862    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-746cdc56fc-xpvd9" podUID="d6b9e911-6989-44f5-a261-79334324488c" containerName="manager" containerID="cri-o://48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682" gracePeriod=30
Jul 08 19:12:56.993071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:56.993044    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:57.084675 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.084645    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert\") pod \"d6b9e911-6989-44f5-a261-79334324488c\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") "
Jul 08 19:12:57.084851 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.084735    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-592qr\" (UniqueName: \"kubernetes.io/projected/d6b9e911-6989-44f5-a261-79334324488c-kube-api-access-592qr\") pod \"d6b9e911-6989-44f5-a261-79334324488c\" (UID: \"d6b9e911-6989-44f5-a261-79334324488c\") "
Jul 08 19:12:57.086908 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.086870    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d6b9e911-6989-44f5-a261-79334324488c" (UID: "d6b9e911-6989-44f5-a261-79334324488c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:12:57.087008 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.086946    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b9e911-6989-44f5-a261-79334324488c-kube-api-access-592qr" (OuterVolumeSpecName: "kube-api-access-592qr") pod "d6b9e911-6989-44f5-a261-79334324488c" (UID: "d6b9e911-6989-44f5-a261-79334324488c"). InnerVolumeSpecName "kube-api-access-592qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:12:57.186218 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.186140    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-592qr\" (UniqueName: \"kubernetes.io/projected/d6b9e911-6989-44f5-a261-79334324488c-kube-api-access-592qr\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:12:57.186218 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.186169    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6b9e911-6989-44f5-a261-79334324488c-webhook-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:12:57.705100 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.705064    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6b9e911-6989-44f5-a261-79334324488c" containerID="48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682" exitCode=0
Jul 08 19:12:57.705579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.705106    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-746cdc56fc-xpvd9"
Jul 08 19:12:57.705579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.705153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-746cdc56fc-xpvd9" event={"ID":"d6b9e911-6989-44f5-a261-79334324488c","Type":"ContainerDied","Data":"48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682"}
Jul 08 19:12:57.705579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.705213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-746cdc56fc-xpvd9" event={"ID":"d6b9e911-6989-44f5-a261-79334324488c","Type":"ContainerDied","Data":"ec9e06ff6d1684a0b7d8e0d71eb9103259d9b958016844ed0ff90d947b7912d2"}
Jul 08 19:12:57.705579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.705249    2563 scope.go:117] "RemoveContainer" containerID="48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682"
Jul 08 19:12:57.714478 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.714462    2563 scope.go:117] "RemoveContainer" containerID="48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682"
Jul 08 19:12:57.714769 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:12:57.714751    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682\": container with ID starting with 48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682 not found: ID does not exist" containerID="48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682"
Jul 08 19:12:57.714842 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.714775    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682"} err="failed to get container status \"48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682\": rpc error: code = NotFound desc = could not find container \"48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682\": container with ID starting with 48bfe795c970b8eddf1767642d2a5738e9f8bb27f25e217c200566d6e572a682 not found: ID does not exist"
Jul 08 19:12:57.727444 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.727422    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-746cdc56fc-xpvd9"]
Jul 08 19:12:57.732695 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:57.732534    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-746cdc56fc-xpvd9"]
Jul 08 19:12:58.454964 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:12:58.454930    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b9e911-6989-44f5-a261-79334324488c" path="/var/lib/kubelet/pods/d6b9e911-6989-44f5-a261-79334324488c/volumes"
Jul 08 19:13:15.738940 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:15.738903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:16.231807 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.231724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:16.281752 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.281717    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-786f787578-wg5t9"]
Jul 08 19:13:16.282088 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.282074    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b9e911-6989-44f5-a261-79334324488c" containerName="manager"
Jul 08 19:13:16.282156 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.282089    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b9e911-6989-44f5-a261-79334324488c" containerName="manager"
Jul 08 19:13:16.282156 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.282148    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b9e911-6989-44f5-a261-79334324488c" containerName="manager"
Jul 08 19:13:16.297435 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.297404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-786f787578-wg5t9"]
Jul 08 19:13:16.297615 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.297510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.300746 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.300717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 08 19:13:16.300887 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.300731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9gbm5\""
Jul 08 19:13:16.343858 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.343826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5581d8e-be71-4cf6-9395-ff69e3559166-maas-api-tls\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.344022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.343920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5581d8e-be71-4cf6-9395-ff69e3559166-openshift-service-ca\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.344022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.343976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kkrh\" (UniqueName: \"kubernetes.io/projected/d5581d8e-be71-4cf6-9395-ff69e3559166-kube-api-access-6kkrh\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.370522 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.370486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:16.444882 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.444846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5581d8e-be71-4cf6-9395-ff69e3559166-maas-api-tls\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.445060 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.444907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5581d8e-be71-4cf6-9395-ff69e3559166-openshift-service-ca\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.445060 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.444943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kkrh\" (UniqueName: \"kubernetes.io/projected/d5581d8e-be71-4cf6-9395-ff69e3559166-kube-api-access-6kkrh\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.445503 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.445480    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5581d8e-be71-4cf6-9395-ff69e3559166-openshift-service-ca\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.447285 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.447263    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5581d8e-be71-4cf6-9395-ff69e3559166-maas-api-tls\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.455260 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.455233    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kkrh\" (UniqueName: \"kubernetes.io/projected/d5581d8e-be71-4cf6-9395-ff69e3559166-kube-api-access-6kkrh\") pod \"maas-api-786f787578-wg5t9\" (UID: \"d5581d8e-be71-4cf6-9395-ff69e3559166\") " pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.607904 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.607865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:16.684841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.684811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:16.754557 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.754533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-786f787578-wg5t9"]
Jul 08 19:13:16.756756 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:13:16.756723    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5581d8e_be71_4cf6_9395_ff69e3559166.slice/crio-8d122685c4641c91dd758bab5a4af565978c993f36203d7e56762f6b867171c9 WatchSource:0}: Error finding container 8d122685c4641c91dd758bab5a4af565978c993f36203d7e56762f6b867171c9: Status 404 returned error can't find the container with id 8d122685c4641c91dd758bab5a4af565978c993f36203d7e56762f6b867171c9
Jul 08 19:13:16.762220 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.757822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:16.798605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.798565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9" event={"ID":"d5581d8e-be71-4cf6-9395-ff69e3559166","Type":"ContainerStarted","Data":"8d122685c4641c91dd758bab5a4af565978c993f36203d7e56762f6b867171c9"}
Jul 08 19:13:16.802228 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.802185    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-58t7x"]
Jul 08 19:13:16.806806 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.806788    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:16.810014 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.809990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 08 19:13:16.816786 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.816759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-58t7x"]
Jul 08 19:13:16.821823 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.821801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sm57r"]
Jul 08 19:13:16.825314 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.825297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:16.829089 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.829070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-589gg\""
Jul 08 19:13:16.838969 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.838945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sm57r"]
Jul 08 19:13:16.949011 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.948923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180632f-51cb-4843-a11d-76de044d5d8c-plugins-config-volume\") pod \"payload-processing-58f86c7798-sm57r\" (UID: \"5180632f-51cb-4843-a11d-76de044d5d8c\") " pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:16.949182 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.949067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlp5\" (UniqueName: \"kubernetes.io/projected/f886fde6-1a29-4db8-8596-5f9ca1d94034-kube-api-access-4wlp5\") pod \"payload-pre-processing-7964457696-58t7x\" (UID: \"f886fde6-1a29-4db8-8596-5f9ca1d94034\") " pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:16.949182 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.949128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6tp\" (UniqueName: \"kubernetes.io/projected/5180632f-51cb-4843-a11d-76de044d5d8c-kube-api-access-7t6tp\") pod \"payload-processing-58f86c7798-sm57r\" (UID: \"5180632f-51cb-4843-a11d-76de044d5d8c\") " pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:16.949182 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:16.949171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f886fde6-1a29-4db8-8596-5f9ca1d94034-plugins-config-volume\") pod \"payload-pre-processing-7964457696-58t7x\" (UID: \"f886fde6-1a29-4db8-8596-5f9ca1d94034\") " pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:17.050089 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.050045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlp5\" (UniqueName: \"kubernetes.io/projected/f886fde6-1a29-4db8-8596-5f9ca1d94034-kube-api-access-4wlp5\") pod \"payload-pre-processing-7964457696-58t7x\" (UID: \"f886fde6-1a29-4db8-8596-5f9ca1d94034\") " pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:17.050089 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.050095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6tp\" (UniqueName: \"kubernetes.io/projected/5180632f-51cb-4843-a11d-76de044d5d8c-kube-api-access-7t6tp\") pod \"payload-processing-58f86c7798-sm57r\" (UID: \"5180632f-51cb-4843-a11d-76de044d5d8c\") " pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:17.050340 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.050119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f886fde6-1a29-4db8-8596-5f9ca1d94034-plugins-config-volume\") pod \"payload-pre-processing-7964457696-58t7x\" (UID: \"f886fde6-1a29-4db8-8596-5f9ca1d94034\") " pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:17.050340 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.050188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180632f-51cb-4843-a11d-76de044d5d8c-plugins-config-volume\") pod \"payload-processing-58f86c7798-sm57r\" (UID: \"5180632f-51cb-4843-a11d-76de044d5d8c\") " pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:17.050961 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.050934    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f886fde6-1a29-4db8-8596-5f9ca1d94034-plugins-config-volume\") pod \"payload-pre-processing-7964457696-58t7x\" (UID: \"f886fde6-1a29-4db8-8596-5f9ca1d94034\") " pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:17.051062 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.050961    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5180632f-51cb-4843-a11d-76de044d5d8c-plugins-config-volume\") pod \"payload-processing-58f86c7798-sm57r\" (UID: \"5180632f-51cb-4843-a11d-76de044d5d8c\") " pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:17.063656 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.063615    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlp5\" (UniqueName: \"kubernetes.io/projected/f886fde6-1a29-4db8-8596-5f9ca1d94034-kube-api-access-4wlp5\") pod \"payload-pre-processing-7964457696-58t7x\" (UID: \"f886fde6-1a29-4db8-8596-5f9ca1d94034\") " pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:17.063982 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.063964    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6tp\" (UniqueName: \"kubernetes.io/projected/5180632f-51cb-4843-a11d-76de044d5d8c-kube-api-access-7t6tp\") pod \"payload-processing-58f86c7798-sm57r\" (UID: \"5180632f-51cb-4843-a11d-76de044d5d8c\") " pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:17.118288 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.118250    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:17.135873 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.135846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:17.272978 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.272944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-58t7x"]
Jul 08 19:13:17.275210 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:13:17.275171    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf886fde6_1a29_4db8_8596_5f9ca1d94034.slice/crio-ce06b1f1c96f9e2c288317ece7ae0b7a5e3a9e63b79e0f518f339867db31f163 WatchSource:0}: Error finding container ce06b1f1c96f9e2c288317ece7ae0b7a5e3a9e63b79e0f518f339867db31f163: Status 404 returned error can't find the container with id ce06b1f1c96f9e2c288317ece7ae0b7a5e3a9e63b79e0f518f339867db31f163
Jul 08 19:13:17.289051 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.289029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sm57r"]
Jul 08 19:13:17.291852 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:13:17.291825    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5180632f_51cb_4843_a11d_76de044d5d8c.slice/crio-7649c39dee22207a69fead7a17d15f9668ec8e8ec0a6e51809d8cbe9b58a34e2 WatchSource:0}: Error finding container 7649c39dee22207a69fead7a17d15f9668ec8e8ec0a6e51809d8cbe9b58a34e2: Status 404 returned error can't find the container with id 7649c39dee22207a69fead7a17d15f9668ec8e8ec0a6e51809d8cbe9b58a34e2
Jul 08 19:13:17.806579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.806500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sm57r" event={"ID":"5180632f-51cb-4843-a11d-76de044d5d8c","Type":"ContainerStarted","Data":"7649c39dee22207a69fead7a17d15f9668ec8e8ec0a6e51809d8cbe9b58a34e2"}
Jul 08 19:13:17.807933 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:17.807898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x" event={"ID":"f886fde6-1a29-4db8-8596-5f9ca1d94034","Type":"ContainerStarted","Data":"ce06b1f1c96f9e2c288317ece7ae0b7a5e3a9e63b79e0f518f339867db31f163"}
Jul 08 19:13:19.818965 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.818928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x" event={"ID":"f886fde6-1a29-4db8-8596-5f9ca1d94034","Type":"ContainerStarted","Data":"cf8f0a3746945c2821034e4d757b0de0c7bcbd0eac93dc95e3fe4e7af611e88b"}
Jul 08 19:13:19.819454 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.819400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:19.821078 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.820997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9" event={"ID":"d5581d8e-be71-4cf6-9395-ff69e3559166","Type":"ContainerStarted","Data":"5e016396368abfc07449d4e88c9236b5a83ec50f0d5b842c3e7ba7f794a2ddc0"}
Jul 08 19:13:19.821194 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.821086    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:19.822425 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.822401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sm57r" event={"ID":"5180632f-51cb-4843-a11d-76de044d5d8c","Type":"ContainerStarted","Data":"f01868759c5b9cbbebe71b7ae06d4b0660328643102e8b92d6c6cf141f7d86d8"}
Jul 08 19:13:19.822525 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.822514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:19.844175 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.844115    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x" podStartSLOduration=1.825003932 podStartE2EDuration="3.844083037s" totalImagesPullingTime="2.019079105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:19.840960884 +0000 UTC m=+858.484981615" watchObservedRunningTime="2026-07-08 19:13:19.844083037 +0000 UTC m=+858.488103769"
Jul 08 19:13:19.882326 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.882207    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-sm57r" podStartSLOduration=1.87730533 podStartE2EDuration="3.8821872s" totalImagesPullingTime="2.00488187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:19.862766122 +0000 UTC m=+858.506786854" watchObservedRunningTime="2026-07-08 19:13:19.8821872 +0000 UTC m=+858.526207933"
Jul 08 19:13:19.882992 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:19.882963    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9" podStartSLOduration=1.345512405 podStartE2EDuration="3.882951119s" totalImagesPullingTime="2.537438714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:19.881066951 +0000 UTC m=+858.525087680" watchObservedRunningTime="2026-07-08 19:13:19.882951119 +0000 UTC m=+858.526971849"
Jul 08 19:13:24.848680 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:24.848649    2563 generic.go:356] "Generic (PLEG): container finished" podID="8dd6eb0d-e25c-4fb5-9a14-c85c50626e61" containerID="f98b2c7e537fc36ca16ac416d89420a8e1f09da192af091e4ca0f2d5fd97c49f" exitCode=0
Jul 08 19:13:24.849107 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:24.848723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" event={"ID":"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61","Type":"ContainerDied","Data":"f98b2c7e537fc36ca16ac416d89420a8e1f09da192af091e4ca0f2d5fd97c49f"}
Jul 08 19:13:24.849152 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:24.849115    2563 scope.go:117] "RemoveContainer" containerID="f98b2c7e537fc36ca16ac416d89420a8e1f09da192af091e4ca0f2d5fd97c49f"
Jul 08 19:13:24.850200 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:24.850172    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:13:24.850312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:24.850249    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:13:24.850312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:24.850299    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:13:25.832524 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:25.832495    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-786f787578-wg5t9"
Jul 08 19:13:25.874402 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:25.874366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" event={"ID":"8dd6eb0d-e25c-4fb5-9a14-c85c50626e61","Type":"ContainerStarted","Data":"517d13050fab884db469fb809fcf297b79bcfb765386024d0c6e41545e821188"}
Jul 08 19:13:25.964814 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:25.964769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:13:25.966202 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:25.966173    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cdrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 08 19:13:25.966343 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:25.966236    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" podUID="8dd6eb0d-e25c-4fb5-9a14-c85c50626e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 08 19:13:26.364820 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:26.364729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:13:26.844894 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:26.844808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:26.945832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:26.945794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:26.965727 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:26.965692    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cdrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 08 19:13:26.965913 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:26.965748    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" podUID="8dd6eb0d-e25c-4fb5-9a14-c85c50626e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 08 19:13:27.965296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:27.965208    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cdrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 08 19:13:27.965296 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:27.965287    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" podUID="8dd6eb0d-e25c-4fb5-9a14-c85c50626e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 08 19:13:28.965905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:28.965870    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cdrm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 08 19:13:28.966319 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:28.965957    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2" podUID="8dd6eb0d-e25c-4fb5-9a14-c85c50626e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 08 19:13:29.969014 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:29.968982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:13:29.970011 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:29.969990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cdrm2"
Jul 08 19:13:30.829283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:30.829247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x" podUID="f886fde6-1a29-4db8-8596-5f9ca1d94034" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.42:9004: connect: connection refused"
Jul 08 19:13:30.829512 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:30.829493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-sm57r"
Jul 08 19:13:33.802721 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.802687    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"]
Jul 08 19:13:33.805764 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.805743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:33.808776 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.808754    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 08 19:13:33.810409 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.810385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 08 19:13:33.810731 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.810432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 08 19:13:33.810917 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.810481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hs4tk\""
Jul 08 19:13:33.813722 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.813704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"]
Jul 08 19:13:33.830063 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.830033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:33.909567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.909526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:33.909567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.909569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:33.909787 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.909684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:33.909787 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.909737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgqw\" (UniqueName: \"kubernetes.io/projected/3a0bf640-b2e7-4cef-82ed-ba29425c1037-kube-api-access-dsgqw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:33.909787 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.909783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:33.909897 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:33.909864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0bf640-b2e7-4cef-82ed-ba29425c1037-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.010806 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.010773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgqw\" (UniqueName: \"kubernetes.io/projected/3a0bf640-b2e7-4cef-82ed-ba29425c1037-kube-api-access-dsgqw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.010977 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.010816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.010977 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.010865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0bf640-b2e7-4cef-82ed-ba29425c1037-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.010977 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.010900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.010977 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.010966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.011176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.011016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.011280 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.011258    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.011345 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.011311    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.011405 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.011359    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.013201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.013179    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a0bf640-b2e7-4cef-82ed-ba29425c1037-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.013463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.013445    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a0bf640-b2e7-4cef-82ed-ba29425c1037-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.019721 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.019697    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgqw\" (UniqueName: \"kubernetes.io/projected/3a0bf640-b2e7-4cef-82ed-ba29425c1037-kube-api-access-dsgqw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7\" (UID: \"3a0bf640-b2e7-4cef-82ed-ba29425c1037\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.116991 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.116908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:34.251171 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.251146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"]
Jul 08 19:13:34.252786 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:13:34.252764    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0bf640_b2e7_4cef_82ed_ba29425c1037.slice/crio-3ee8233c0d0ff942610931e9a0fc352536ffc5406249b2164e2a8ea1ba0ac81f WatchSource:0}: Error finding container 3ee8233c0d0ff942610931e9a0fc352536ffc5406249b2164e2a8ea1ba0ac81f: Status 404 returned error can't find the container with id 3ee8233c0d0ff942610931e9a0fc352536ffc5406249b2164e2a8ea1ba0ac81f
Jul 08 19:13:34.923330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:34.923294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7" event={"ID":"3a0bf640-b2e7-4cef-82ed-ba29425c1037","Type":"ContainerStarted","Data":"3ee8233c0d0ff942610931e9a0fc352536ffc5406249b2164e2a8ea1ba0ac81f"}
Jul 08 19:13:35.931382 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:35.931344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7" event={"ID":"3a0bf640-b2e7-4cef-82ed-ba29425c1037","Type":"ContainerStarted","Data":"6d1b226f2a48895a114413529b0be70e0350e61f459125fa158b9452700138c6"}
Jul 08 19:13:35.951161 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:35.951115    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7" podStartSLOduration=1.9582832510000001 podStartE2EDuration="2.95109937s" totalImagesPullingTime="992.816119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:35.950055912 +0000 UTC m=+874.594076645" watchObservedRunningTime="2026-07-08 19:13:35.95109937 +0000 UTC m=+874.595120100"
Jul 08 19:13:40.828049 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:40.828018    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-58t7x"
Jul 08 19:13:44.117357 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:44.117320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:44.117782 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:44.117387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:44.130101 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:44.130076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:44.991021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:44.990984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7"
Jul 08 19:13:45.293823 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.293741    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"]
Jul 08 19:13:45.296361 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.296338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.299336 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.299316    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 08 19:13:45.306023 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.305998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.306138 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.306037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.306208 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.306147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3416c93-1c30-4f6a-b27e-ef432104bc4d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.306260 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.306247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.306311 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.306289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlb4\" (UniqueName: \"kubernetes.io/projected/f3416c93-1c30-4f6a-b27e-ef432104bc4d-kube-api-access-cnlb4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.306369 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.306342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.311439 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.311413    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"]
Jul 08 19:13:45.325811 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.325777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:45.407274 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3416c93-1c30-4f6a-b27e-ef432104bc4d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407419 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407419 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlb4\" (UniqueName: \"kubernetes.io/projected/f3416c93-1c30-4f6a-b27e-ef432104bc4d-kube-api-access-cnlb4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407419 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407419 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407419 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407889 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407859    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407886    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.407989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.407910    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.409647 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.409613    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3416c93-1c30-4f6a-b27e-ef432104bc4d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.409828 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.409812    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3416c93-1c30-4f6a-b27e-ef432104bc4d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.415866 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.415843    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlb4\" (UniqueName: \"kubernetes.io/projected/f3416c93-1c30-4f6a-b27e-ef432104bc4d-kube-api-access-cnlb4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9rnwt\" (UID: \"f3416c93-1c30-4f6a-b27e-ef432104bc4d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.607148 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.607057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:45.741998 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.741974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"]
Jul 08 19:13:45.743981 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:13:45.743955    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3416c93_1c30_4f6a_b27e_ef432104bc4d.slice/crio-61c3ead9e8795571afad1bc1b4186e53f05773cfc24f5ba191685574a6503ed9 WatchSource:0}: Error finding container 61c3ead9e8795571afad1bc1b4186e53f05773cfc24f5ba191685574a6503ed9: Status 404 returned error can't find the container with id 61c3ead9e8795571afad1bc1b4186e53f05773cfc24f5ba191685574a6503ed9
Jul 08 19:13:45.985218 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:45.985192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt" event={"ID":"f3416c93-1c30-4f6a-b27e-ef432104bc4d","Type":"ContainerStarted","Data":"61c3ead9e8795571afad1bc1b4186e53f05773cfc24f5ba191685574a6503ed9"}
Jul 08 19:13:46.991711 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:46.991674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt" event={"ID":"f3416c93-1c30-4f6a-b27e-ef432104bc4d","Type":"ContainerStarted","Data":"5a9709072ebf7a15c42f7de9d1a846ea82b224df81450f54a63d9e7ae6f27d46"}
Jul 08 19:13:47.012676 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:47.012612    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt" podStartSLOduration=1.823235588 podStartE2EDuration="2.012597562s" totalImagesPullingTime="189.361974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:47.010439526 +0000 UTC m=+885.654460257" watchObservedRunningTime="2026-07-08 19:13:47.012597562 +0000 UTC m=+885.656618292"
Jul 08 19:13:55.607508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:55.607419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:55.607508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:55.607467    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:55.620147 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:55.620123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:56.047370 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:56.047341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9rnwt"
Jul 08 19:13:56.457114 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:56.457031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:57.951563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:57.951528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:13:58.708686 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:13:58.708653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:14:02.237826 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:02.237798    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:14:02.244563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:02.244540    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:14:02.266469 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:02.266446    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:14:02.274483 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:02.274454    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:14:08.199825 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.199791    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"]
Jul 08 19:14:08.203514 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.203496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.206541 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.206519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 08 19:14:08.218147 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.218117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"]
Jul 08 19:14:08.222196 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.222171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:14:08.319616 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.319583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.319791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.319661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.319791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.319681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71be099e-7c18-4fd9-bdde-199d3f998c7a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.319791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.319717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.319791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.319756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8r7\" (UniqueName: \"kubernetes.io/projected/71be099e-7c18-4fd9-bdde-199d3f998c7a-kube-api-access-vx8r7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.319933 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.319826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.420993 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.420953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421169 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421169 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71be099e-7c18-4fd9-bdde-199d3f998c7a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421169 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421169 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8r7\" (UniqueName: \"kubernetes.io/projected/71be099e-7c18-4fd9-bdde-199d3f998c7a-kube-api-access-vx8r7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421392 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421400    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421409    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.421569 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.421526    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.423453 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.423432    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71be099e-7c18-4fd9-bdde-199d3f998c7a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.423645 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.423600    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71be099e-7c18-4fd9-bdde-199d3f998c7a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.430637 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.430603    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8r7\" (UniqueName: \"kubernetes.io/projected/71be099e-7c18-4fd9-bdde-199d3f998c7a-kube-api-access-vx8r7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn\" (UID: \"71be099e-7c18-4fd9-bdde-199d3f998c7a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.519482 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.519442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:08.653067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.653039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"]
Jul 08 19:14:08.654168 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:14:08.654135    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71be099e_7c18_4fd9_bdde_199d3f998c7a.slice/crio-8a23db15b3abd27a92d91340c51c59065c57e0a2b10a0baaa49773ee7142e228 WatchSource:0}: Error finding container 8a23db15b3abd27a92d91340c51c59065c57e0a2b10a0baaa49773ee7142e228: Status 404 returned error can't find the container with id 8a23db15b3abd27a92d91340c51c59065c57e0a2b10a0baaa49773ee7142e228
Jul 08 19:14:08.655544 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:08.655525    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 19:14:09.107684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:09.107576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn" event={"ID":"71be099e-7c18-4fd9-bdde-199d3f998c7a","Type":"ContainerStarted","Data":"2f2eb8fd1af533bc48a06565b792a1c1c2abcf6c15df216b1b35ec49b16ab34b"}
Jul 08 19:14:09.107684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:09.107670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn" event={"ID":"71be099e-7c18-4fd9-bdde-199d3f998c7a","Type":"ContainerStarted","Data":"8a23db15b3abd27a92d91340c51c59065c57e0a2b10a0baaa49773ee7142e228"}
Jul 08 19:14:09.129002 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:09.128953    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn" podStartSLOduration=0.935420185 podStartE2EDuration="1.128934877s" totalImagesPullingTime="193.514692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:14:09.127434509 +0000 UTC m=+907.771455239" watchObservedRunningTime="2026-07-08 19:14:09.128934877 +0000 UTC m=+907.772955604"
Jul 08 19:14:18.520464 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:18.520420    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:18.520909 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:18.520502    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:18.533225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:18.533203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:19.172441 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:19.172412    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn"
Jul 08 19:14:24.249048 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:24.249007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:14:27.941682 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:27.941648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:14:43.682909 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:14:43.682867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:15:00.150820 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.150787    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"]
Jul 08 19:15:00.154038 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.154022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"
Jul 08 19:15:00.163300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.163275    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"]
Jul 08 19:15:00.295710 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.295677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4c8\" (UniqueName: \"kubernetes.io/projected/70ae4f01-1953-4c9f-9ea0-b87d206c00ef-kube-api-access-rb4c8\") pod \"maas-api-key-cleanup-29725635-5hhzk\" (UID: \"70ae4f01-1953-4c9f-9ea0-b87d206c00ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"
Jul 08 19:15:00.396475 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.396432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4c8\" (UniqueName: \"kubernetes.io/projected/70ae4f01-1953-4c9f-9ea0-b87d206c00ef-kube-api-access-rb4c8\") pod \"maas-api-key-cleanup-29725635-5hhzk\" (UID: \"70ae4f01-1953-4c9f-9ea0-b87d206c00ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"
Jul 08 19:15:00.406602 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.406535    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4c8\" (UniqueName: \"kubernetes.io/projected/70ae4f01-1953-4c9f-9ea0-b87d206c00ef-kube-api-access-rb4c8\") pod \"maas-api-key-cleanup-29725635-5hhzk\" (UID: \"70ae4f01-1953-4c9f-9ea0-b87d206c00ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"
Jul 08 19:15:00.466067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.466034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"
Jul 08 19:15:00.806680 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:00.806653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"]
Jul 08 19:15:00.807811 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:15:00.807781    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ae4f01_1953_4c9f_9ea0_b87d206c00ef.slice/crio-571492c72f4fcc36d155ae3af53263ece5ce830b00c58bcfbb5574a3a5cd0e7b WatchSource:0}: Error finding container 571492c72f4fcc36d155ae3af53263ece5ce830b00c58bcfbb5574a3a5cd0e7b: Status 404 returned error can't find the container with id 571492c72f4fcc36d155ae3af53263ece5ce830b00c58bcfbb5574a3a5cd0e7b
Jul 08 19:15:01.379926 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:01.379893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk" event={"ID":"70ae4f01-1953-4c9f-9ea0-b87d206c00ef","Type":"ContainerStarted","Data":"571492c72f4fcc36d155ae3af53263ece5ce830b00c58bcfbb5574a3a5cd0e7b"}
Jul 08 19:15:04.398369 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:04.398325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk" event={"ID":"70ae4f01-1953-4c9f-9ea0-b87d206c00ef","Type":"ContainerStarted","Data":"a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562"}
Jul 08 19:15:04.421116 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:04.421065    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk" podStartSLOduration=1.912290741 podStartE2EDuration="4.421051601s" totalImagesPullingTime="2.50876086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:04.419133704 +0000 UTC m=+963.063154434" watchObservedRunningTime="2026-07-08 19:15:04.421051601 +0000 UTC m=+963.065072330"
Jul 08 19:15:24.910584 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:24.910505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:15:25.025704 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:25.025668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:15:26.163793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:26.163764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:15:26.257098 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:26.257062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:15:46.979514 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:15:46.979480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:16:00.011629 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:00.011579    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"]
Jul 08 19:16:00.012098 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:00.011842    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk" podUID="70ae4f01-1953-4c9f-9ea0-b87d206c00ef" containerName="cleanup" containerID="cri-o://a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562" gracePeriod=30
Jul 08 19:16:30.188241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.188209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"
Jul 08 19:16:30.300555 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.300527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb4c8\" (UniqueName: \"kubernetes.io/projected/70ae4f01-1953-4c9f-9ea0-b87d206c00ef-kube-api-access-rb4c8\") pod \"70ae4f01-1953-4c9f-9ea0-b87d206c00ef\" (UID: \"70ae4f01-1953-4c9f-9ea0-b87d206c00ef\") "
Jul 08 19:16:30.302811 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.302771    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ae4f01-1953-4c9f-9ea0-b87d206c00ef-kube-api-access-rb4c8" (OuterVolumeSpecName: "kube-api-access-rb4c8") pod "70ae4f01-1953-4c9f-9ea0-b87d206c00ef" (UID: "70ae4f01-1953-4c9f-9ea0-b87d206c00ef"). InnerVolumeSpecName "kube-api-access-rb4c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:16:30.402237 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.402155    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4c8\" (UniqueName: \"kubernetes.io/projected/70ae4f01-1953-4c9f-9ea0-b87d206c00ef-kube-api-access-rb4c8\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:16:30.840421 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.840380    2563 generic.go:356] "Generic (PLEG): container finished" podID="70ae4f01-1953-4c9f-9ea0-b87d206c00ef" containerID="a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562" exitCode=137
Jul 08 19:16:30.840597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.840423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk" event={"ID":"70ae4f01-1953-4c9f-9ea0-b87d206c00ef","Type":"ContainerDied","Data":"a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562"}
Jul 08 19:16:30.840597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.840430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"
Jul 08 19:16:30.840597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.840477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk" event={"ID":"70ae4f01-1953-4c9f-9ea0-b87d206c00ef","Type":"ContainerDied","Data":"571492c72f4fcc36d155ae3af53263ece5ce830b00c58bcfbb5574a3a5cd0e7b"}
Jul 08 19:16:30.840597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.840505    2563 scope.go:117] "RemoveContainer" containerID="a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562"
Jul 08 19:16:30.849276 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.849253    2563 scope.go:117] "RemoveContainer" containerID="a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562"
Jul 08 19:16:30.849603 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:16:30.849584    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562\": container with ID starting with a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562 not found: ID does not exist" containerID="a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562"
Jul 08 19:16:30.849670 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.849611    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562"} err="failed to get container status \"a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562\": rpc error: code = NotFound desc = could not find container \"a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562\": container with ID starting with a7215e8d9e308bd59e54722d10fec25e1f1103179bb84a72d65232173ae08562 not found: ID does not exist"
Jul 08 19:16:30.860703 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.860671    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"]
Jul 08 19:16:30.863817 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:30.863796    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725635-5hhzk"]
Jul 08 19:16:32.455752 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:32.455715    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ae4f01-1953-4c9f-9ea0-b87d206c00ef" path="/var/lib/kubelet/pods/70ae4f01-1953-4c9f-9ea0-b87d206c00ef/volumes"
Jul 08 19:16:36.604860 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:36.604825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:16:36.720400 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:36.720356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-7ngcr"]
Jul 08 19:16:36.720602 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:36.720564    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr" podUID="24ab442b-1fce-409a-bc8e-51c3ca8707e3" containerName="manager" containerID="cri-o://4b56904e06d80e3916bd97851367b4434b466ff08531e234a791c9bb72fd672f" gracePeriod=30
Jul 08 19:16:36.874197 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:36.874113    2563 generic.go:356] "Generic (PLEG): container finished" podID="24ab442b-1fce-409a-bc8e-51c3ca8707e3" containerID="4b56904e06d80e3916bd97851367b4434b466ff08531e234a791c9bb72fd672f" exitCode=0
Jul 08 19:16:36.874329 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:36.874184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr" event={"ID":"24ab442b-1fce-409a-bc8e-51c3ca8707e3","Type":"ContainerDied","Data":"4b56904e06d80e3916bd97851367b4434b466ff08531e234a791c9bb72fd672f"}
Jul 08 19:16:36.968371 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:36.968347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:16:37.063468 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.063435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24ab442b-1fce-409a-bc8e-51c3ca8707e3-webhook-cert\") pod \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") "
Jul 08 19:16:37.063669 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.063529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9jt9\" (UniqueName: \"kubernetes.io/projected/24ab442b-1fce-409a-bc8e-51c3ca8707e3-kube-api-access-s9jt9\") pod \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\" (UID: \"24ab442b-1fce-409a-bc8e-51c3ca8707e3\") "
Jul 08 19:16:37.065534 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.065502    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ab442b-1fce-409a-bc8e-51c3ca8707e3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "24ab442b-1fce-409a-bc8e-51c3ca8707e3" (UID: "24ab442b-1fce-409a-bc8e-51c3ca8707e3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:16:37.065671 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.065539    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ab442b-1fce-409a-bc8e-51c3ca8707e3-kube-api-access-s9jt9" (OuterVolumeSpecName: "kube-api-access-s9jt9") pod "24ab442b-1fce-409a-bc8e-51c3ca8707e3" (UID: "24ab442b-1fce-409a-bc8e-51c3ca8707e3"). InnerVolumeSpecName "kube-api-access-s9jt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:16:37.165300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.165213    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24ab442b-1fce-409a-bc8e-51c3ca8707e3-webhook-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:16:37.165300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.165244    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jt9\" (UniqueName: \"kubernetes.io/projected/24ab442b-1fce-409a-bc8e-51c3ca8707e3-kube-api-access-s9jt9\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:16:37.881670 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.881640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr"
Jul 08 19:16:37.882118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.881683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fcfb68c87-7ngcr" event={"ID":"24ab442b-1fce-409a-bc8e-51c3ca8707e3","Type":"ContainerDied","Data":"cf26fa02c1768c91532165384828ba346ec8c2ade4aa2b2b2a6dd8a15de3d298"}
Jul 08 19:16:37.882118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.881745    2563 scope.go:117] "RemoveContainer" containerID="4b56904e06d80e3916bd97851367b4434b466ff08531e234a791c9bb72fd672f"
Jul 08 19:16:37.904388 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.904361    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-7ngcr"]
Jul 08 19:16:37.907886 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:37.907865    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-7ngcr"]
Jul 08 19:16:38.455509 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.455478    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ab442b-1fce-409a-bc8e-51c3ca8707e3" path="/var/lib/kubelet/pods/24ab442b-1fce-409a-bc8e-51c3ca8707e3/volumes"
Jul 08 19:16:38.607497 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.607465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:16:38.759212 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.759181    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-j72rp"]
Jul 08 19:16:38.759532 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.759520    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ab442b-1fce-409a-bc8e-51c3ca8707e3" containerName="manager"
Jul 08 19:16:38.759579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.759534    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ab442b-1fce-409a-bc8e-51c3ca8707e3" containerName="manager"
Jul 08 19:16:38.759579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.759550    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ae4f01-1953-4c9f-9ea0-b87d206c00ef" containerName="cleanup"
Jul 08 19:16:38.759579 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.759555    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ae4f01-1953-4c9f-9ea0-b87d206c00ef" containerName="cleanup"
Jul 08 19:16:38.759695 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.759614    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ae4f01-1953-4c9f-9ea0-b87d206c00ef" containerName="cleanup"
Jul 08 19:16:38.759695 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.759642    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ab442b-1fce-409a-bc8e-51c3ca8707e3" containerName="manager"
Jul 08 19:16:38.763890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.763873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:38.769899 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.769874    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 08 19:16:38.770046 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.769877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vkzct\""
Jul 08 19:16:38.770046 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.769877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 08 19:16:38.776589 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.776559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-j72rp"]
Jul 08 19:16:38.881896 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.881862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglmb\" (UniqueName: \"kubernetes.io/projected/8b33789c-a165-491d-8f9f-d9ceeabe6150-kube-api-access-kglmb\") pod \"maas-controller-7fcfb68c87-j72rp\" (UID: \"8b33789c-a165-491d-8f9f-d9ceeabe6150\") " pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:38.882303 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.881970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b33789c-a165-491d-8f9f-d9ceeabe6150-webhook-cert\") pod \"maas-controller-7fcfb68c87-j72rp\" (UID: \"8b33789c-a165-491d-8f9f-d9ceeabe6150\") " pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:38.983391 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.983354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b33789c-a165-491d-8f9f-d9ceeabe6150-webhook-cert\") pod \"maas-controller-7fcfb68c87-j72rp\" (UID: \"8b33789c-a165-491d-8f9f-d9ceeabe6150\") " pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:38.983520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.983445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglmb\" (UniqueName: \"kubernetes.io/projected/8b33789c-a165-491d-8f9f-d9ceeabe6150-kube-api-access-kglmb\") pod \"maas-controller-7fcfb68c87-j72rp\" (UID: \"8b33789c-a165-491d-8f9f-d9ceeabe6150\") " pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:38.985776 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.985755    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b33789c-a165-491d-8f9f-d9ceeabe6150-webhook-cert\") pod \"maas-controller-7fcfb68c87-j72rp\" (UID: \"8b33789c-a165-491d-8f9f-d9ceeabe6150\") " pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:38.992974 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:38.992951    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglmb\" (UniqueName: \"kubernetes.io/projected/8b33789c-a165-491d-8f9f-d9ceeabe6150-kube-api-access-kglmb\") pod \"maas-controller-7fcfb68c87-j72rp\" (UID: \"8b33789c-a165-491d-8f9f-d9ceeabe6150\") " pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:39.073872 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:39.073776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:39.206419 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:39.206396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fcfb68c87-j72rp"]
Jul 08 19:16:39.208077 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:16:39.208052    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b33789c_a165_491d_8f9f_d9ceeabe6150.slice/crio-aba78fbc33531c6371c375eb024759ce1d3d2651ed3adc3963a39f9fd3d25fc4 WatchSource:0}: Error finding container aba78fbc33531c6371c375eb024759ce1d3d2651ed3adc3963a39f9fd3d25fc4: Status 404 returned error can't find the container with id aba78fbc33531c6371c375eb024759ce1d3d2651ed3adc3963a39f9fd3d25fc4
Jul 08 19:16:39.894097 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:39.894066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fcfb68c87-j72rp" event={"ID":"8b33789c-a165-491d-8f9f-d9ceeabe6150","Type":"ContainerStarted","Data":"7a345598db74c2537dee832bc0895d4985fb2cb51e81f44116dc6d1d7f201a0b"}
Jul 08 19:16:39.894503 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:39.894115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fcfb68c87-j72rp" event={"ID":"8b33789c-a165-491d-8f9f-d9ceeabe6150","Type":"ContainerStarted","Data":"aba78fbc33531c6371c375eb024759ce1d3d2651ed3adc3963a39f9fd3d25fc4"}
Jul 08 19:16:39.894503 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:39.894146    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:39.913355 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:39.913304    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7fcfb68c87-j72rp" podStartSLOduration=1.335720611 podStartE2EDuration="1.913288369s" totalImagesPullingTime="577.567758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:39.913114855 +0000 UTC m=+1058.557135585" watchObservedRunningTime="2026-07-08 19:16:39.913288369 +0000 UTC m=+1058.557309099"
Jul 08 19:16:48.024990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:48.024957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:16:50.428490 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:50.428454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:16:50.904910 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:50.904875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7fcfb68c87-j72rp"
Jul 08 19:16:56.410002 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:16:56.409921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:21.306909 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:21.306858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:21.409310 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:21.409269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:21.540960 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:21.540928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:27.690921 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:27.690890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:27.754605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:27.754570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:27.859099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:27.859063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:27.957831 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:27.957748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:28.058218 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:28.058178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:34.461018 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:34.460974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:17:34.563350 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:17:34.563319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:19:02.272825 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:19:02.272800    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:19:02.279793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:19:02.279773    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:19:02.299596 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:19:02.299577    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:19:02.306409 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:19:02.306390    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:20:22.464808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:20:22.464728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:20:24.042871 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:20:24.042832    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:20:50.845771 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:20:50.845733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:20:50.946490 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:20:50.946458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:22:24.781375 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:22:24.781341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:23:59.558443 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:23:59.558411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:24:00.259318 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:24:00.259276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:24:02.307489 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:24:02.307388    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:24:02.315005 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:24:02.314987    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:24:02.330411 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:24:02.330391    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:24:02.336850 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:24:02.336832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:28:10.848845 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:10.848795    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:20.491493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:20.491460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:28.761554 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:28.761516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:33.893123 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:33.893086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:36.840247 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:36.840206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:36.931215 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:36.931182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:37.290695 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:37.290658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:44.252298 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.252261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:44.354147 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.354111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:44.475745 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.475715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"]
Jul 08 19:28:44.479128 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.479111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.482571 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.482551    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dd9ee87b-openshift-default-dockercfg-78bdx\""
Jul 08 19:28:44.494913 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.494886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:44.513167 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.513098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"]
Jul 08 19:28:44.561229 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istiod-ca-cert\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561394 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-envoy\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561394 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-podinfo\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561394 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-certs\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxqp\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-kube-api-access-qsxqp\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561508 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-data\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561577 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-credential-socket\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561577 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-token\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.561668 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.561604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-socket\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.591422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.591371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:44.662526 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istiod-ca-cert\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-envoy\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-podinfo\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-certs\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662709 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxqp\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-kube-api-access-qsxqp\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-data\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-credential-socket\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-token\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.662905 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.662823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-socket\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.663104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.663070    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-certs\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.663157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.663070    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-data\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.663428 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.663402    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-socket\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.663516 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.663432    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-credential-socket\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.663676 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.663659    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istiod-ca-cert\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.664919 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.664887    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-envoy\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.665056 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.665040    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-podinfo\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.672523 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.672485    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxqp\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-kube-api-access-qsxqp\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.672691 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.672671    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-token\") pod \"e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") " pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.789585 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.789511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:44.929037 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.929002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"]
Jul 08 19:28:44.929209 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:28:44.929189    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4b9cbb_5197_4bb8_81a8_b100857f0997.slice/crio-9ff02cc7ace0ddc39662bd440fa7fdc0fca40a253c2a9462c9aaa4114f14f025 WatchSource:0}: Error finding container 9ff02cc7ace0ddc39662bd440fa7fdc0fca40a253c2a9462c9aaa4114f14f025: Status 404 returned error can't find the container with id 9ff02cc7ace0ddc39662bd440fa7fdc0fca40a253c2a9462c9aaa4114f14f025
Jul 08 19:28:44.931353 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.931316    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:28:44.931479 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.931382    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:28:44.931479 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:44.931413    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:28:45.600792 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.600749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld" event={"ID":"5e4b9cbb-5197-4bb8-81a8-b100857f0997","Type":"ContainerStarted","Data":"d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc"}
Jul 08 19:28:45.601168 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.600838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld" event={"ID":"5e4b9cbb-5197-4bb8-81a8-b100857f0997","Type":"ContainerStarted","Data":"9ff02cc7ace0ddc39662bd440fa7fdc0fca40a253c2a9462c9aaa4114f14f025"}
Jul 08 19:28:45.637670 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.637596    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld" podStartSLOduration=1.637575335 podStartE2EDuration="1.637575335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:45.63623601 +0000 UTC m=+1784.280256751" watchObservedRunningTime="2026-07-08 19:28:45.637575335 +0000 UTC m=+1784.281596065"
Jul 08 19:28:45.688641 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.688580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:45.775575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.775532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:45.776131 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.776107    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"]
Jul 08 19:28:45.779776 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.779755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.784035 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.783988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dd9ee87b-serving-cert\""
Jul 08 19:28:45.789893 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.789870    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:45.792855 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.792832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"]
Jul 08 19:28:45.796489 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.796469    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:45.863136 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.863054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:45.876140 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.876112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c32d16-ea63-406d-9e79-05821cc8e9f3-openshift-service-ca\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.876278 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.876169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.876340 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.876270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxxf\" (UniqueName: \"kubernetes.io/projected/d9c32d16-ea63-406d-9e79-05821cc8e9f3-kube-api-access-dvxxf\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.977493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.977460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.977717 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.977532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxxf\" (UniqueName: \"kubernetes.io/projected/d9c32d16-ea63-406d-9e79-05821cc8e9f3-kube-api-access-dvxxf\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.977717 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:28:45.977618    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-serving-cert: secret "maas-api-e2e-ait-dd9ee87b-serving-cert" not found
Jul 08 19:28:45.977859 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.977649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c32d16-ea63-406d-9e79-05821cc8e9f3-openshift-service-ca\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.977859 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:28:45.977720    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls podName:d9c32d16-ea63-406d-9e79-05821cc8e9f3 nodeName:}" failed. No retries permitted until 2026-07-08 19:28:46.477696583 +0000 UTC m=+1785.121717309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls") pod "maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n" (UID: "d9c32d16-ea63-406d-9e79-05821cc8e9f3") : secret "maas-api-e2e-ait-dd9ee87b-serving-cert" not found
Jul 08 19:28:45.978276 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.978257    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c32d16-ea63-406d-9e79-05821cc8e9f3-openshift-service-ca\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:45.992780 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:45.992755    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxxf\" (UniqueName: \"kubernetes.io/projected/d9c32d16-ea63-406d-9e79-05821cc8e9f3-kube-api-access-dvxxf\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:46.204722 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.204606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:46.417352 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.417319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:46.482384 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.482297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:46.484706 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.484683    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls\") pod \"maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:46.576518 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.576484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:46.608808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.608771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:46.609677 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.609647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:46.692935 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.692902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:46.845828 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.845801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"]
Jul 08 19:28:46.846573 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:28:46.846531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c32d16_ea63_406d_9e79_05821cc8e9f3.slice/crio-026c79ca7bee8359e9f9ba133ede01ab92904fef655867abed8893d747d9a011 WatchSource:0}: Error finding container 026c79ca7bee8359e9f9ba133ede01ab92904fef655867abed8893d747d9a011: Status 404 returned error can't find the container with id 026c79ca7bee8359e9f9ba133ede01ab92904fef655867abed8893d747d9a011
Jul 08 19:28:46.848153 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:46.848136    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 19:28:47.615139 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:47.615106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n" event={"ID":"d9c32d16-ea63-406d-9e79-05821cc8e9f3","Type":"ContainerStarted","Data":"d3c481efb9e37daff7607b62c2700db29b996e60a3c0a4ef6e6304b3d21089a1"}
Jul 08 19:28:47.615618 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:47.615160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n" event={"ID":"d9c32d16-ea63-406d-9e79-05821cc8e9f3","Type":"ContainerStarted","Data":"026c79ca7bee8359e9f9ba133ede01ab92904fef655867abed8893d747d9a011"}
Jul 08 19:28:47.615618 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:47.615450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:47.639839 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:47.639794    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n" podStartSLOduration=2.094777849 podStartE2EDuration="2.639778682s" totalImagesPullingTime="545.000833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:47.63690291 +0000 UTC m=+1786.280923640" watchObservedRunningTime="2026-07-08 19:28:47.639778682 +0000 UTC m=+1786.283799412"
Jul 08 19:28:48.267839 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:48.267793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:50.828145 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:50.828104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:51.050325 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.050295    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"]
Jul 08 19:28:51.050845 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.050812    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld" podUID="5e4b9cbb-5197-4bb8-81a8-b100857f0997" containerName="istio-proxy" containerID="cri-o://d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc" gracePeriod=30
Jul 08 19:28:51.067982 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.067949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:51.180198 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.180107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:51.373467 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.373434    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"]
Jul 08 19:28:51.373691 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.373665    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n" podUID="d9c32d16-ea63-406d-9e79-05821cc8e9f3" containerName="maas-api" containerID="cri-o://d3c481efb9e37daff7607b62c2700db29b996e60a3c0a4ef6e6304b3d21089a1" gracePeriod=30
Jul 08 19:28:51.389940 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.389914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:51.660849 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.660687    2563 generic.go:356] "Generic (PLEG): container finished" podID="d9c32d16-ea63-406d-9e79-05821cc8e9f3" containerID="d3c481efb9e37daff7607b62c2700db29b996e60a3c0a4ef6e6304b3d21089a1" exitCode=0
Jul 08 19:28:51.660849 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.660744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n" event={"ID":"d9c32d16-ea63-406d-9e79-05821cc8e9f3","Type":"ContainerDied","Data":"d3c481efb9e37daff7607b62c2700db29b996e60a3c0a4ef6e6304b3d21089a1"}
Jul 08 19:28:51.660849 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.660796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n" event={"ID":"d9c32d16-ea63-406d-9e79-05821cc8e9f3","Type":"ContainerDied","Data":"026c79ca7bee8359e9f9ba133ede01ab92904fef655867abed8893d747d9a011"}
Jul 08 19:28:51.660849 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.660821    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="026c79ca7bee8359e9f9ba133ede01ab92904fef655867abed8893d747d9a011"
Jul 08 19:28:51.664009 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.663986    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:51.728918 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.728889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls\") pod \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") "
Jul 08 19:28:51.729081 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.728923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvxxf\" (UniqueName: \"kubernetes.io/projected/d9c32d16-ea63-406d-9e79-05821cc8e9f3-kube-api-access-dvxxf\") pod \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") "
Jul 08 19:28:51.729081 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.729013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c32d16-ea63-406d-9e79-05821cc8e9f3-openshift-service-ca\") pod \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\" (UID: \"d9c32d16-ea63-406d-9e79-05821cc8e9f3\") "
Jul 08 19:28:51.729359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.729338    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c32d16-ea63-406d-9e79-05821cc8e9f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9c32d16-ea63-406d-9e79-05821cc8e9f3" (UID: "d9c32d16-ea63-406d-9e79-05821cc8e9f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:28:51.730928 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.730900    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9c32d16-ea63-406d-9e79-05821cc8e9f3" (UID: "d9c32d16-ea63-406d-9e79-05821cc8e9f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:28:51.731031 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.731010    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c32d16-ea63-406d-9e79-05821cc8e9f3-kube-api-access-dvxxf" (OuterVolumeSpecName: "kube-api-access-dvxxf") pod "d9c32d16-ea63-406d-9e79-05821cc8e9f3" (UID: "d9c32d16-ea63-406d-9e79-05821cc8e9f3"). InnerVolumeSpecName "kube-api-access-dvxxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:28:51.829817 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.829730    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9c32d16-ea63-406d-9e79-05821cc8e9f3-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:51.829817 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.829759    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxxf\" (UniqueName: \"kubernetes.io/projected/d9c32d16-ea63-406d-9e79-05821cc8e9f3-kube-api-access-dvxxf\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:51.829817 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:51.829769    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c32d16-ea63-406d-9e79-05821cc8e9f3-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:52.666799 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:52.666766    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"
Jul 08 19:28:52.685764 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:52.685737    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"]
Jul 08 19:28:52.690912 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:52.690885    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dd9ee87b-544df74db9-zx24n"]
Jul 08 19:28:52.832950 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:52.832914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:54.456087 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:54.456056    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c32d16-ea63-406d-9e79-05821cc8e9f3" path="/var/lib/kubelet/pods/d9c32d16-ea63-406d-9e79-05821cc8e9f3/volumes"
Jul 08 19:28:57.300535 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.300508    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:57.386742 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.386675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istiod-ca-cert\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.386742 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.386715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsxqp\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-kube-api-access-qsxqp\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.386911 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.386747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-envoy\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.386911 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.386792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-podinfo\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.386996 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.386928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-certs\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.386996 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.386989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-socket\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.387104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-credential-socket\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.387104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-data\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.387104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387043    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:28:57.387104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-token\") pod \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\" (UID: \"5e4b9cbb-5197-4bb8-81a8-b100857f0997\") "
Jul 08 19:28:57.387307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387223    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:28:57.387371 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387322    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:28:57.387511 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387479    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:28:57.387511 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387489    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.387654 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387525    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.387654 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387541    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.387863 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.387843    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:28:57.389677 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.389619    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:28:57.389677 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.389655    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:28:57.389829 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.389716    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:28:57.389829 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.389720    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-kube-api-access-qsxqp" (OuterVolumeSpecName: "kube-api-access-qsxqp") pod "5e4b9cbb-5197-4bb8-81a8-b100857f0997" (UID: "5e4b9cbb-5197-4bb8-81a8-b100857f0997"). InnerVolumeSpecName "kube-api-access-qsxqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:28:57.488395 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.488361    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.488395 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.488389    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.488395 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.488401    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.488649 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.488414    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.488649 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.488427    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxqp\" (UniqueName: \"kubernetes.io/projected/5e4b9cbb-5197-4bb8-81a8-b100857f0997-kube-api-access-qsxqp\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.488649 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.488439    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e4b9cbb-5197-4bb8-81a8-b100857f0997-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:28:57.694812 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.694723    2563 generic.go:356] "Generic (PLEG): container finished" podID="5e4b9cbb-5197-4bb8-81a8-b100857f0997" containerID="d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc" exitCode=0
Jul 08 19:28:57.694812 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.694772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"
Jul 08 19:28:57.694978 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.694813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld" event={"ID":"5e4b9cbb-5197-4bb8-81a8-b100857f0997","Type":"ContainerDied","Data":"d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc"}
Jul 08 19:28:57.694978 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.694865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld" event={"ID":"5e4b9cbb-5197-4bb8-81a8-b100857f0997","Type":"ContainerDied","Data":"9ff02cc7ace0ddc39662bd440fa7fdc0fca40a253c2a9462c9aaa4114f14f025"}
Jul 08 19:28:57.694978 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.694889    2563 scope.go:117] "RemoveContainer" containerID="d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc"
Jul 08 19:28:57.704693 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.704677    2563 scope.go:117] "RemoveContainer" containerID="d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc"
Jul 08 19:28:57.704935 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:28:57.704919    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc\": container with ID starting with d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc not found: ID does not exist" containerID="d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc"
Jul 08 19:28:57.704974 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.704943    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc"} err="failed to get container status \"d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc\": rpc error: code = NotFound desc = could not find container \"d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc\": container with ID starting with d4cb0099145a801d11af312e8d61cdee7d2b6d141af2c210b00d1164812580fc not found: ID does not exist"
Jul 08 19:28:57.721114 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.721088    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"]
Jul 08 19:28:57.723312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.723292    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dd9ee87b-openshift-default-bc77b8595-fqcld"]
Jul 08 19:28:57.746954 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.746919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:57.825964 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.825935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:57.921372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.921332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"]
Jul 08 19:28:57.921808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.921789    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c32d16-ea63-406d-9e79-05821cc8e9f3" containerName="maas-api"
Jul 08 19:28:57.921808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.921810    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c32d16-ea63-406d-9e79-05821cc8e9f3" containerName="maas-api"
Jul 08 19:28:57.921973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.921836    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4b9cbb-5197-4bb8-81a8-b100857f0997" containerName="istio-proxy"
Jul 08 19:28:57.921973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.921844    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4b9cbb-5197-4bb8-81a8-b100857f0997" containerName="istio-proxy"
Jul 08 19:28:57.921973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.921924    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c32d16-ea63-406d-9e79-05821cc8e9f3" containerName="maas-api"
Jul 08 19:28:57.921973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.921938    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4b9cbb-5197-4bb8-81a8-b100857f0997" containerName="istio-proxy"
Jul 08 19:28:57.925659 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.925611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.928463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.928444    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6a6cbfe4-openshift-default-dockercfg-shz6d\""
Jul 08 19:28:57.950131 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.950066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"]
Jul 08 19:28:57.992762 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-certs\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.992886 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-envoy\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.992886 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-credential-socket\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.992886 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wjn\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-kube-api-access-48wjn\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.993067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-data\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.993067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-podinfo\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.993067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-socket\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.993067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.992998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-token\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:57.993219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:57.993066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istiod-ca-cert\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.002752 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.002728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:58.093441 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:58.093593 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-envoy\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.093593 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-credential-socket\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.093593 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wjn\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-kube-api-access-48wjn\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.093801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-data\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.093801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-podinfo\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.093801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-socket\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.093801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-token\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.093801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istiod-ca-cert\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.094065 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-certs\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.094065 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093910    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-credential-socket\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.094065 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.093979    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-data\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.094230 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.094126    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-socket\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.094230 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.094144    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-certs\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.094651 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.094608    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istiod-ca-cert\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.096539 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.096521    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-podinfo\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.096606 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.096540    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-envoy\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.108398 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.108372    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-token\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.108559 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.108541    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wjn\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-kube-api-access-48wjn\") pod \"e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") " pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.236418 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.236349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:58.373914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.373889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"]
Jul 08 19:28:58.375812 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:28:58.375786    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ee9b1e_9b14_4806_bea6_cc7d6697b1c4.slice/crio-16c68f5cd7c0d4fbad6e7b29ffb11666e3dfeaf9903764eff089f4e9dd1f461c WatchSource:0}: Error finding container 16c68f5cd7c0d4fbad6e7b29ffb11666e3dfeaf9903764eff089f4e9dd1f461c: Status 404 returned error can't find the container with id 16c68f5cd7c0d4fbad6e7b29ffb11666e3dfeaf9903764eff089f4e9dd1f461c
Jul 08 19:28:58.378112 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.378074    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:28:58.378195 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.378177    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:28:58.378241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.378226    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:28:58.457167 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.457133    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e4b9cbb-5197-4bb8-81a8-b100857f0997" path="/var/lib/kubelet/pods/5e4b9cbb-5197-4bb8-81a8-b100857f0997/volumes"
Jul 08 19:28:58.702493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.702456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx" event={"ID":"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4","Type":"ContainerStarted","Data":"18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3"}
Jul 08 19:28:58.702702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.702513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx" event={"ID":"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4","Type":"ContainerStarted","Data":"16c68f5cd7c0d4fbad6e7b29ffb11666e3dfeaf9903764eff089f4e9dd1f461c"}
Jul 08 19:28:58.760006 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:58.759960    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx" podStartSLOduration=1.759944626 podStartE2EDuration="1.759944626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:28:58.756984565 +0000 UTC m=+1797.401005296" watchObservedRunningTime="2026-07-08 19:28:58.759944626 +0000 UTC m=+1797.403965430"
Jul 08 19:28:59.137153 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.137119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:59.237073 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.237040    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"]
Jul 08 19:28:59.241167 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.241141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:59.241317 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.241247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.242743 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.242725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:59.244390 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.244370    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6a6cbfe4-serving-cert\""
Jul 08 19:28:59.254518 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.254490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:59.258425 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.258405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"]
Jul 08 19:28:59.305126 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.305103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwx95\" (UniqueName: \"kubernetes.io/projected/32c47958-55e0-423c-a547-21a88a6638cc-kube-api-access-vwx95\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.305247 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.305172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c47958-55e0-423c-a547-21a88a6638cc-openshift-service-ca\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.305247 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.305242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.361840 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.361762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:59.406826 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.406736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.406826 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.406785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwx95\" (UniqueName: \"kubernetes.io/projected/32c47958-55e0-423c-a547-21a88a6638cc-kube-api-access-vwx95\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.407309 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.406828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c47958-55e0-423c-a547-21a88a6638cc-openshift-service-ca\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.407309 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:28:59.406880    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-serving-cert: secret "maas-api-e2e-ait-6a6cbfe4-serving-cert" not found
Jul 08 19:28:59.407309 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:28:59.406941    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls podName:32c47958-55e0-423c-a547-21a88a6638cc nodeName:}" failed. No retries permitted until 2026-07-08 19:28:59.906923009 +0000 UTC m=+1798.550943734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls") pod "maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh" (UID: "32c47958-55e0-423c-a547-21a88a6638cc") : secret "maas-api-e2e-ait-6a6cbfe4-serving-cert" not found
Jul 08 19:28:59.407501 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.407481    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c47958-55e0-423c-a547-21a88a6638cc-openshift-service-ca\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.416808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.416788    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwx95\" (UniqueName: \"kubernetes.io/projected/32c47958-55e0-423c-a547-21a88a6638cc-kube-api-access-vwx95\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.707556 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.707475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:59.708541 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.708523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:28:59.764844 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.764811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:28:59.912401 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.912373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.914767 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.914738    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls\") pod \"maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:28:59.922673 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:28:59.922647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:00.022823 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:00.022788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:00.152670 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:00.152618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:29:00.290815 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:00.290784    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"]
Jul 08 19:29:00.291399 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:29:00.291373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c47958_55e0_423c_a547_21a88a6638cc.slice/crio-0ccc7e9a7be995b71d414a361ee781505afe65457358e4d83e817123a1153960 WatchSource:0}: Error finding container 0ccc7e9a7be995b71d414a361ee781505afe65457358e4d83e817123a1153960: Status 404 returned error can't find the container with id 0ccc7e9a7be995b71d414a361ee781505afe65457358e4d83e817123a1153960
Jul 08 19:29:00.714349 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:00.714320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh" event={"ID":"32c47958-55e0-423c-a547-21a88a6638cc","Type":"ContainerStarted","Data":"e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3"}
Jul 08 19:29:00.714688 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:00.714511    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:29:00.714688 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:00.714554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh" event={"ID":"32c47958-55e0-423c-a547-21a88a6638cc","Type":"ContainerStarted","Data":"0ccc7e9a7be995b71d414a361ee781505afe65457358e4d83e817123a1153960"}
Jul 08 19:29:00.736763 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:00.736719    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh" podStartSLOduration=1.414071686 podStartE2EDuration="1.736704697s" totalImagesPullingTime="322.633011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:00.734923191 +0000 UTC m=+1799.378943923" watchObservedRunningTime="2026-07-08 19:29:00.736704697 +0000 UTC m=+1799.380725446"
Jul 08 19:29:01.292247 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:01.292205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:02.342292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:02.342265    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:29:02.349462 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:02.349439    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:29:02.362340 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:02.362321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:29:02.368720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:02.368703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:29:04.343598 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:04.343549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:04.969648 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:04.969242    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"]
Jul 08 19:29:04.969648 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:04.969495    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh" podUID="32c47958-55e0-423c-a547-21a88a6638cc" containerName="maas-api" containerID="cri-o://e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3" gracePeriod=30
Jul 08 19:29:04.981691 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:04.981606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:05.270411 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.270384    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:29:05.369080 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.369047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c47958-55e0-423c-a547-21a88a6638cc-openshift-service-ca\") pod \"32c47958-55e0-423c-a547-21a88a6638cc\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") "
Jul 08 19:29:05.369520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.369096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwx95\" (UniqueName: \"kubernetes.io/projected/32c47958-55e0-423c-a547-21a88a6638cc-kube-api-access-vwx95\") pod \"32c47958-55e0-423c-a547-21a88a6638cc\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") "
Jul 08 19:29:05.369520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.369157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls\") pod \"32c47958-55e0-423c-a547-21a88a6638cc\" (UID: \"32c47958-55e0-423c-a547-21a88a6638cc\") "
Jul 08 19:29:05.369520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.369378    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c47958-55e0-423c-a547-21a88a6638cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32c47958-55e0-423c-a547-21a88a6638cc" (UID: "32c47958-55e0-423c-a547-21a88a6638cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:29:05.369671 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.369536    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c47958-55e0-423c-a547-21a88a6638cc-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:05.371366 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.371342    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c47958-55e0-423c-a547-21a88a6638cc-kube-api-access-vwx95" (OuterVolumeSpecName: "kube-api-access-vwx95") pod "32c47958-55e0-423c-a547-21a88a6638cc" (UID: "32c47958-55e0-423c-a547-21a88a6638cc"). InnerVolumeSpecName "kube-api-access-vwx95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:29:05.371467 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.371382    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32c47958-55e0-423c-a547-21a88a6638cc" (UID: "32c47958-55e0-423c-a547-21a88a6638cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:29:05.470306 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.470276    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c47958-55e0-423c-a547-21a88a6638cc-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:05.470306 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.470304    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwx95\" (UniqueName: \"kubernetes.io/projected/32c47958-55e0-423c-a547-21a88a6638cc-kube-api-access-vwx95\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:05.743866 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.743822    2563 generic.go:356] "Generic (PLEG): container finished" podID="32c47958-55e0-423c-a547-21a88a6638cc" containerID="e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3" exitCode=0
Jul 08 19:29:05.744073 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.744014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh" event={"ID":"32c47958-55e0-423c-a547-21a88a6638cc","Type":"ContainerDied","Data":"e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3"}
Jul 08 19:29:05.744144 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.744070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh" event={"ID":"32c47958-55e0-423c-a547-21a88a6638cc","Type":"ContainerDied","Data":"0ccc7e9a7be995b71d414a361ee781505afe65457358e4d83e817123a1153960"}
Jul 08 19:29:05.744144 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.744102    2563 scope.go:117] "RemoveContainer" containerID="e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3"
Jul 08 19:29:05.744292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.744271    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"
Jul 08 19:29:05.765214 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.764749    2563 scope.go:117] "RemoveContainer" containerID="e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3"
Jul 08 19:29:05.765214 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:29:05.765121    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3\": container with ID starting with e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3 not found: ID does not exist" containerID="e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3"
Jul 08 19:29:05.765214 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.765160    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3"} err="failed to get container status \"e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3\": rpc error: code = NotFound desc = could not find container \"e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3\": container with ID starting with e96bf07eec7df8d6745ce57a21a867cdf0cc8c4f87536a54173cbdc7974e1ff3 not found: ID does not exist"
Jul 08 19:29:05.783308 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.783284    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"]
Jul 08 19:29:05.786054 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:05.786030    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a6cbfe4-5d6c76854-c6jgh"]
Jul 08 19:29:06.458670 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:06.458613    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c47958-55e0-423c-a547-21a88a6638cc" path="/var/lib/kubelet/pods/32c47958-55e0-423c-a547-21a88a6638cc/volumes"
Jul 08 19:29:10.493991 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:10.493959    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"]
Jul 08 19:29:10.494416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:10.494138    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx" podUID="85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" containerName="istio-proxy" containerID="cri-o://18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3" gracePeriod=30
Jul 08 19:29:10.525234 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:10.525200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:10.625307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:10.625266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:10.725965 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:10.725931    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:11.548990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.548967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:29:11.630475 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-socket\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.630643 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-envoy\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.630643 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-podinfo\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.630789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630641    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48wjn\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-kube-api-access-48wjn\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.630789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-certs\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.630789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istiod-ca-cert\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.630789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-credential-socket\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.631015 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-data\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.631015 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.630836    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-token\") pod \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\" (UID: \"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4\") "
Jul 08 19:29:11.631338 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.631156    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:11.631338 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.631259    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.631716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.631661    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:11.631716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.631685    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:29:11.631891 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.631752    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:11.631956 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.631825    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:11.633791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.633754    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:29:11.633900 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.633790    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-kube-api-access-48wjn" (OuterVolumeSpecName: "kube-api-access-48wjn") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "kube-api-access-48wjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:29:11.633900 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.633873    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:29:11.634368 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.634339    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" (UID: "85ee9b1e-9b14-4806-bea6-cc7d6697b1c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:11.732310 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732280    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.732310 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732305    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.732465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732315    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48wjn\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-kube-api-access-48wjn\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.732465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732326    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.732465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732334    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.732465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732342    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.732465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732350    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.732465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.732357    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:11.784439 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.784409    2563 generic.go:356] "Generic (PLEG): container finished" podID="85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" containerID="18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3" exitCode=0
Jul 08 19:29:11.784564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.784453    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"
Jul 08 19:29:11.784564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.784495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx" event={"ID":"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4","Type":"ContainerDied","Data":"18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3"}
Jul 08 19:29:11.784564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.784557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx" event={"ID":"85ee9b1e-9b14-4806-bea6-cc7d6697b1c4","Type":"ContainerDied","Data":"16c68f5cd7c0d4fbad6e7b29ffb11666e3dfeaf9903764eff089f4e9dd1f461c"}
Jul 08 19:29:11.784749 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.784587    2563 scope.go:117] "RemoveContainer" containerID="18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3"
Jul 08 19:29:11.794408 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.794383    2563 scope.go:117] "RemoveContainer" containerID="18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3"
Jul 08 19:29:11.794675 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:29:11.794655    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3\": container with ID starting with 18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3 not found: ID does not exist" containerID="18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3"
Jul 08 19:29:11.794738 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.794683    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3"} err="failed to get container status \"18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3\": rpc error: code = NotFound desc = could not find container \"18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3\": container with ID starting with 18a448ff2b405703c4ad7d8d36479c07948d139f4a726c9e090c4be1c5c18bb3 not found: ID does not exist"
Jul 08 19:29:11.810330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.810304    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"]
Jul 08 19:29:11.815349 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:11.815331    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6a6cbfe4-openshift-default-55f5d458c7-2d9mx"]
Jul 08 19:29:12.160870 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:12.160839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:12.455649 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:12.455557    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" path="/var/lib/kubelet/pods/85ee9b1e-9b14-4806-bea6-cc7d6697b1c4/volumes"
Jul 08 19:29:16.804014 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:16.803975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:16.895237 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:16.895204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:17.102293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.102263    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"]
Jul 08 19:29:17.102617 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.102603    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" containerName="istio-proxy"
Jul 08 19:29:17.102617 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.102617    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" containerName="istio-proxy"
Jul 08 19:29:17.102772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.102653    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c47958-55e0-423c-a547-21a88a6638cc" containerName="maas-api"
Jul 08 19:29:17.102772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.102659    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c47958-55e0-423c-a547-21a88a6638cc" containerName="maas-api"
Jul 08 19:29:17.102772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.102732    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c47958-55e0-423c-a547-21a88a6638cc" containerName="maas-api"
Jul 08 19:29:17.102772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.102742    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ee9b1e-9b14-4806-bea6-cc7d6697b1c4" containerName="istio-proxy"
Jul 08 19:29:17.107324 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.107301    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.113793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.113688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1ec97c42-openshift-default-dockercfg-mnqpw\""
Jul 08 19:29:17.163541 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.163508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:17.222277 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.222242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"]
Jul 08 19:29:17.285594 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-token\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-socket\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-podinfo\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-certs\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwq9p\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-kube-api-access-fwq9p\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285898 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-credential-socket\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285898 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istiod-ca-cert\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285898 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-envoy\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.285898 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.285892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-data\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.386867 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.386777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-certs\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.386867 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.386825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwq9p\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-kube-api-access-fwq9p\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.386867 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.386846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-credential-socket\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.386887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istiod-ca-cert\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.386909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-envoy\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.386941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-data\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.386980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-token\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.387026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-socket\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387157 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.387050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-podinfo\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387448 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.387212    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-certs\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387448 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.387409    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-socket\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.387461    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-data\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.387535    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-credential-socket\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.387687 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.387665    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istiod-ca-cert\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.389222 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.389200    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-envoy\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.389401 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.389384    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-podinfo\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.401782 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.401760    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwq9p\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-kube-api-access-fwq9p\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.405585 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.405563    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-token\") pod \"e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") " pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.407325 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.407300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:17.417038 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.417007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:17.564206 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.564184    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"]
Jul 08 19:29:17.565684 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:29:17.565657    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b7c3c9_d8cb_4872_ae40_ea4327422e36.slice/crio-1d49ca2f2e49a3e43e74545bc19dee897a5004b39b5b7cd1d3263fb5f3e94636 WatchSource:0}: Error finding container 1d49ca2f2e49a3e43e74545bc19dee897a5004b39b5b7cd1d3263fb5f3e94636: Status 404 returned error can't find the container with id 1d49ca2f2e49a3e43e74545bc19dee897a5004b39b5b7cd1d3263fb5f3e94636
Jul 08 19:29:17.567864 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.567829    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:29:17.567936 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.567889    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:29:17.567974 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.567937    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:29:17.817151 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.817114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s" event={"ID":"f4b7c3c9-d8cb-4872-ae40-ea4327422e36","Type":"ContainerStarted","Data":"c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc"}
Jul 08 19:29:17.817525 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.817180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s" event={"ID":"f4b7c3c9-d8cb-4872-ae40-ea4327422e36","Type":"ContainerStarted","Data":"1d49ca2f2e49a3e43e74545bc19dee897a5004b39b5b7cd1d3263fb5f3e94636"}
Jul 08 19:29:17.843485 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:17.843432    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s" podStartSLOduration=0.843411662 podStartE2EDuration="843.411662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:17.841452179 +0000 UTC m=+1816.485472910" watchObservedRunningTime="2026-07-08 19:29:17.843411662 +0000 UTC m=+1816.487432395"
Jul 08 19:29:18.308790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.308746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:18.394515 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.394477    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"]
Jul 08 19:29:18.398774 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.398752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.401821 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.401798    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1ec97c42-serving-cert\""
Jul 08 19:29:18.415326 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.415299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"]
Jul 08 19:29:18.417564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.417545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:18.423402 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.423381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:18.432364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.432335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:18.499719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.499678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spflw\" (UniqueName: \"kubernetes.io/projected/52c10539-9a6c-43c3-a33d-840a146190c5-kube-api-access-spflw\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.499942 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.499753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c10539-9a6c-43c3-a33d-840a146190c5-maas-api-tls\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.499942 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.499830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c10539-9a6c-43c3-a33d-840a146190c5-openshift-service-ca\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.549192 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.549161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:18.601045 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.600966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c10539-9a6c-43c3-a33d-840a146190c5-openshift-service-ca\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.601045 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.601031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spflw\" (UniqueName: \"kubernetes.io/projected/52c10539-9a6c-43c3-a33d-840a146190c5-kube-api-access-spflw\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.601226 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.601065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c10539-9a6c-43c3-a33d-840a146190c5-maas-api-tls\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.601719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.601698    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c10539-9a6c-43c3-a33d-840a146190c5-openshift-service-ca\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.603490 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.603470    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c10539-9a6c-43c3-a33d-840a146190c5-maas-api-tls\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.618298 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.618271    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spflw\" (UniqueName: \"kubernetes.io/projected/52c10539-9a6c-43c3-a33d-840a146190c5-kube-api-access-spflw\") pod \"maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.654501 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.654475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:18.709286 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.709260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:18.824396 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.824366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:18.825539 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.825515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:18.845278 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:18.845259    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"]
Jul 08 19:29:18.845993 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:29:18.845972    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c10539_9a6c_43c3_a33d_840a146190c5.slice/crio-538291c08747faf5dea2e2f54d1b1dde3e6914ecf3663e0344b36de998007415 WatchSource:0}: Error finding container 538291c08747faf5dea2e2f54d1b1dde3e6914ecf3663e0344b36de998007415: Status 404 returned error can't find the container with id 538291c08747faf5dea2e2f54d1b1dde3e6914ecf3663e0344b36de998007415
Jul 08 19:29:19.002122 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:19.002086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:19.104943 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:19.104906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:19.833071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:19.833036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw" event={"ID":"52c10539-9a6c-43c3-a33d-840a146190c5","Type":"ContainerStarted","Data":"7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e"}
Jul 08 19:29:19.833523 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:19.833095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw" event={"ID":"52c10539-9a6c-43c3-a33d-840a146190c5","Type":"ContainerStarted","Data":"538291c08747faf5dea2e2f54d1b1dde3e6914ecf3663e0344b36de998007415"}
Jul 08 19:29:19.833523 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:19.833342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:19.863351 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:19.863301    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw" podStartSLOduration=1.532968823 podStartE2EDuration="1.863285553s" totalImagesPullingTime="330.31673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:29:19.861100492 +0000 UTC m=+1818.505121236" watchObservedRunningTime="2026-07-08 19:29:19.863285553 +0000 UTC m=+1818.507306283"
Jul 08 19:29:20.397490 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:20.397463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:22.734308 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:22.734271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:22.942742 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:22.942684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"]
Jul 08 19:29:22.943042 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:22.943008    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s" podUID="f4b7c3c9-d8cb-4872-ae40-ea4327422e36" containerName="istio-proxy" containerID="cri-o://c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc" gracePeriod=30
Jul 08 19:29:22.972610 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:22.972563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:23.083833 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.083800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:23.302645 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.301614    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"]
Jul 08 19:29:23.302645 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.301901    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw" podUID="52c10539-9a6c-43c3-a33d-840a146190c5" containerName="maas-api" containerID="cri-o://7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e" gracePeriod=30
Jul 08 19:29:23.328771 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.328734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:23.589320 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.589296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:23.661998 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.661968    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spflw\" (UniqueName: \"kubernetes.io/projected/52c10539-9a6c-43c3-a33d-840a146190c5-kube-api-access-spflw\") pod \"52c10539-9a6c-43c3-a33d-840a146190c5\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") "
Jul 08 19:29:23.662192 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.662030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c10539-9a6c-43c3-a33d-840a146190c5-openshift-service-ca\") pod \"52c10539-9a6c-43c3-a33d-840a146190c5\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") "
Jul 08 19:29:23.662192 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.662098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c10539-9a6c-43c3-a33d-840a146190c5-maas-api-tls\") pod \"52c10539-9a6c-43c3-a33d-840a146190c5\" (UID: \"52c10539-9a6c-43c3-a33d-840a146190c5\") "
Jul 08 19:29:23.662395 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.662360    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c10539-9a6c-43c3-a33d-840a146190c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52c10539-9a6c-43c3-a33d-840a146190c5" (UID: "52c10539-9a6c-43c3-a33d-840a146190c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:29:23.664214 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.664189    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c10539-9a6c-43c3-a33d-840a146190c5-kube-api-access-spflw" (OuterVolumeSpecName: "kube-api-access-spflw") pod "52c10539-9a6c-43c3-a33d-840a146190c5" (UID: "52c10539-9a6c-43c3-a33d-840a146190c5"). InnerVolumeSpecName "kube-api-access-spflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:29:23.664313 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.664249    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c10539-9a6c-43c3-a33d-840a146190c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52c10539-9a6c-43c3-a33d-840a146190c5" (UID: "52c10539-9a6c-43c3-a33d-840a146190c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:29:23.763567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.763529    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spflw\" (UniqueName: \"kubernetes.io/projected/52c10539-9a6c-43c3-a33d-840a146190c5-kube-api-access-spflw\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:23.763567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.763557    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c10539-9a6c-43c3-a33d-840a146190c5-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:23.763567 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.763570    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c10539-9a6c-43c3-a33d-840a146190c5-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:23.863028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.862988    2563 generic.go:356] "Generic (PLEG): container finished" podID="52c10539-9a6c-43c3-a33d-840a146190c5" containerID="7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e" exitCode=0
Jul 08 19:29:23.863241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.863179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw" event={"ID":"52c10539-9a6c-43c3-a33d-840a146190c5","Type":"ContainerDied","Data":"7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e"}
Jul 08 19:29:23.863241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.863229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw" event={"ID":"52c10539-9a6c-43c3-a33d-840a146190c5","Type":"ContainerDied","Data":"538291c08747faf5dea2e2f54d1b1dde3e6914ecf3663e0344b36de998007415"}
Jul 08 19:29:23.863365 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.863260    2563 scope.go:117] "RemoveContainer" containerID="7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e"
Jul 08 19:29:23.863443 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.863428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"
Jul 08 19:29:23.874100 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.874081    2563 scope.go:117] "RemoveContainer" containerID="7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e"
Jul 08 19:29:23.874339 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:29:23.874324    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e\": container with ID starting with 7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e not found: ID does not exist" containerID="7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e"
Jul 08 19:29:23.874388 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.874347    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e"} err="failed to get container status \"7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e\": rpc error: code = NotFound desc = could not find container \"7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e\": container with ID starting with 7426792670c6bedcc9ae7fd9119b9937e29b743f755aa4e64dacb8cb9a20b50e not found: ID does not exist"
Jul 08 19:29:23.889218 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.889195    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"]
Jul 08 19:29:23.895267 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:23.895239    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1ec97c42-7b59b7f864-h8kzw"]
Jul 08 19:29:24.408207 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:24.408175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:24.466674 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:24.461385    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c10539-9a6c-43c3-a33d-840a146190c5" path="/var/lib/kubelet/pods/52c10539-9a6c-43c3-a33d-840a146190c5/volumes"
Jul 08 19:29:29.190099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.190043    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:29.316800 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316708    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-token\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.316800 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-credential-socket\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316817    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-envoy\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-data\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istiod-ca-cert\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-certs\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-socket\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.316984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwq9p\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-kube-api-access-fwq9p\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-podinfo\") pod \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\" (UID: \"f4b7c3c9-d8cb-4872-ae40-ea4327422e36\") "
Jul 08 19:29:29.317385 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317099    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:29.317385 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317165    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:29.317385 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317238    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:29.317385 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317279    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:29:29.317564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317408    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.317564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317436    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.317564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317450    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.317564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317463    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.317808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.317733    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:29.318981 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.318954    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:29:29.319239 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.319218    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:29:29.319523 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.319506    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-kube-api-access-fwq9p" (OuterVolumeSpecName: "kube-api-access-fwq9p") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "kube-api-access-fwq9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:29:29.319523 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.319511    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4b7c3c9-d8cb-4872-ae40-ea4327422e36" (UID: "f4b7c3c9-d8cb-4872-ae40-ea4327422e36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:29:29.418357 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.418324    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwq9p\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-kube-api-access-fwq9p\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.418357 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.418355    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.418530 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.418368    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.418530 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.418380    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.418530 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.418390    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4b7c3c9-d8cb-4872-ae40-ea4327422e36-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:29:29.471993 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.471961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:29.574544 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.574468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:29.896832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.896736    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4b7c3c9-d8cb-4872-ae40-ea4327422e36" containerID="c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc" exitCode=0
Jul 08 19:29:29.896832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.896789    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"
Jul 08 19:29:29.897027 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.896826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s" event={"ID":"f4b7c3c9-d8cb-4872-ae40-ea4327422e36","Type":"ContainerDied","Data":"c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc"}
Jul 08 19:29:29.897027 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.896884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s" event={"ID":"f4b7c3c9-d8cb-4872-ae40-ea4327422e36","Type":"ContainerDied","Data":"1d49ca2f2e49a3e43e74545bc19dee897a5004b39b5b7cd1d3263fb5f3e94636"}
Jul 08 19:29:29.897027 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.896906    2563 scope.go:117] "RemoveContainer" containerID="c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc"
Jul 08 19:29:29.906834 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.906818    2563 scope.go:117] "RemoveContainer" containerID="c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc"
Jul 08 19:29:29.907069 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:29:29.907049    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc\": container with ID starting with c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc not found: ID does not exist" containerID="c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc"
Jul 08 19:29:29.907134 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.907076    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc"} err="failed to get container status \"c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc\": rpc error: code = NotFound desc = could not find container \"c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc\": container with ID starting with c122d9f0fac2edeb6cf1d95bc0723e12785d95dc51f5f65cf32bc2fa987122bc not found: ID does not exist"
Jul 08 19:29:29.954335 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.954313    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"]
Jul 08 19:29:29.969154 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:29.969125    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1ec97c42-openshift-default-f9d458899-97d7s"]
Jul 08 19:29:30.021812 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:30.021777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:30.113229 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:30.113184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:30.455432 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:30.455387    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b7c3c9-d8cb-4872-ae40-ea4327422e36" path="/var/lib/kubelet/pods/f4b7c3c9-d8cb-4872-ae40-ea4327422e36/volumes"
Jul 08 19:29:31.629931 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:31.629898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:37.703981 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:37.703942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:37.860271 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:37.860234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:37.960524 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:37.960443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:29:38.439269 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:29:38.439231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:00.150488 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.150448    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"]
Jul 08 19:30:00.150990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.150830    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b7c3c9-d8cb-4872-ae40-ea4327422e36" containerName="istio-proxy"
Jul 08 19:30:00.150990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.150841    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b7c3c9-d8cb-4872-ae40-ea4327422e36" containerName="istio-proxy"
Jul 08 19:30:00.150990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.150850    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c10539-9a6c-43c3-a33d-840a146190c5" containerName="maas-api"
Jul 08 19:30:00.150990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.150855    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c10539-9a6c-43c3-a33d-840a146190c5" containerName="maas-api"
Jul 08 19:30:00.150990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.150916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c10539-9a6c-43c3-a33d-840a146190c5" containerName="maas-api"
Jul 08 19:30:00.150990 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.150927    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b7c3c9-d8cb-4872-ae40-ea4327422e36" containerName="istio-proxy"
Jul 08 19:30:00.154073 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.154056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"
Jul 08 19:30:00.164147 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.164119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"]
Jul 08 19:30:00.299820 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.299786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbtg\" (UniqueName: \"kubernetes.io/projected/d13c1a0e-5124-488a-b079-b2c9c27459f7-kube-api-access-lpbtg\") pod \"maas-api-key-cleanup-29725650-6gr6m\" (UID: \"d13c1a0e-5124-488a-b079-b2c9c27459f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"
Jul 08 19:30:00.400635 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.400532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbtg\" (UniqueName: \"kubernetes.io/projected/d13c1a0e-5124-488a-b079-b2c9c27459f7-kube-api-access-lpbtg\") pod \"maas-api-key-cleanup-29725650-6gr6m\" (UID: \"d13c1a0e-5124-488a-b079-b2c9c27459f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"
Jul 08 19:30:00.411505 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.411474    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbtg\" (UniqueName: \"kubernetes.io/projected/d13c1a0e-5124-488a-b079-b2c9c27459f7-kube-api-access-lpbtg\") pod \"maas-api-key-cleanup-29725650-6gr6m\" (UID: \"d13c1a0e-5124-488a-b079-b2c9c27459f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"
Jul 08 19:30:00.466012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.465980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"
Jul 08 19:30:00.805731 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:00.805700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"]
Jul 08 19:30:00.806887 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:30:00.806864    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13c1a0e_5124_488a_b079_b2c9c27459f7.slice/crio-c25aba5785b6845c0d5ecd627a0b55ba5eb8d32ee72c1f31c2092d575abfc0d8 WatchSource:0}: Error finding container c25aba5785b6845c0d5ecd627a0b55ba5eb8d32ee72c1f31c2092d575abfc0d8: Status 404 returned error can't find the container with id c25aba5785b6845c0d5ecd627a0b55ba5eb8d32ee72c1f31c2092d575abfc0d8
Jul 08 19:30:01.058594 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:01.058516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m" event={"ID":"d13c1a0e-5124-488a-b079-b2c9c27459f7","Type":"ContainerStarted","Data":"c25aba5785b6845c0d5ecd627a0b55ba5eb8d32ee72c1f31c2092d575abfc0d8"}
Jul 08 19:30:02.064951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:02.064917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m" event={"ID":"d13c1a0e-5124-488a-b079-b2c9c27459f7","Type":"ContainerStarted","Data":"129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38"}
Jul 08 19:30:02.083836 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:02.083789    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m" podStartSLOduration=2.083772028 podStartE2EDuration="2.083772028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:02.082600054 +0000 UTC m=+1860.726620797" watchObservedRunningTime="2026-07-08 19:30:02.083772028 +0000 UTC m=+1860.727792786"
Jul 08 19:30:04.448219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:04.448187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:10.676293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:10.676253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:10.799648 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:10.799592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:10.981578 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:10.981488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:26.879965 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:26.879929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:33.393580 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:33.393541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:33.515663 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:33.515599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:44.154017 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:44.153984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:44.957716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:44.957677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:47.655278 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:47.655240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:51.950247 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:51.950207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:52.163662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:52.163608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:52.272494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:52.272449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:53.094426 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:53.094391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:53.234859 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:53.234833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:30:54.974846 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:30:54.974812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:00.012936 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:00.012896    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"]
Jul 08 19:31:00.013443 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:00.013139    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m" podUID="d13c1a0e-5124-488a-b079-b2c9c27459f7" containerName="cleanup" containerID="cri-o://129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38" gracePeriod=30
Jul 08 19:31:01.843825 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:01.843777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:01.952250 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:01.952205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:02.697331 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:02.697291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:09.292837 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:09.292794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:09.396594 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:09.396558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:09.570652 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:09.570557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:09.915172 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:09.915078    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"]
Jul 08 19:31:09.918494 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:09.918473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:09.922674 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:09.922655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-71ba58d9-openshift-default-dockercfg-drv8g\""
Jul 08 19:31:09.932865 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:09.932836    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"]
Jul 08 19:31:10.022858 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.022822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-envoy\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.022858 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.022869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6pw\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-kube-api-access-2g6pw\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.022902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-credential-socket\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.022927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-socket\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.023008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-istio-token\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.023039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-certs\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.023061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-data\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023130 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.023075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecad761-77ff-4775-a354-a8bddcf22520-istio-podinfo\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023455 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.023121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecad761-77ff-4775-a354-a8bddcf22520-istiod-ca-cert\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.023865 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.023836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:10.124713 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.124680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecad761-77ff-4775-a354-a8bddcf22520-istiod-ca-cert\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.124890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.124765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-envoy\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.124890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.124791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6pw\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-kube-api-access-2g6pw\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.124890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.124818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-credential-socket\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.124890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.124848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-socket\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.124932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-istio-token\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.124968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-certs\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.125006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-data\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125118 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.125032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecad761-77ff-4775-a354-a8bddcf22520-istio-podinfo\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125347 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.125216    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-credential-socket\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125347 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.125292    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-socket\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125461 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.125368    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-certs\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125461 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.125394    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecad761-77ff-4775-a354-a8bddcf22520-istiod-ca-cert\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.125562 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.125516    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-data\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.127588 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.127566    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-envoy\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.127768 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.127750    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecad761-77ff-4775-a354-a8bddcf22520-istio-podinfo\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.135862 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.135837    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-istio-token\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.136254 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.136232    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6pw\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-kube-api-access-2g6pw\") pod \"e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") " pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.229088 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.229001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:10.389861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.389816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:10.391701 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.391674    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"]
Jul 08 19:31:10.395608 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:31:10.393708    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecad761_77ff_4775_a354_a8bddcf22520.slice/crio-c146936c47f481a0dfd5212eb022fb5f60745a9a1ca1393e3525a5d7554c5c30 WatchSource:0}: Error finding container c146936c47f481a0dfd5212eb022fb5f60745a9a1ca1393e3525a5d7554c5c30: Status 404 returned error can't find the container with id c146936c47f481a0dfd5212eb022fb5f60745a9a1ca1393e3525a5d7554c5c30
Jul 08 19:31:10.397441 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.397412    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:10.397544 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.397488    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:10.397544 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.397530    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:10.408836 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:10.408808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" event={"ID":"1ecad761-77ff-4775-a354-a8bddcf22520","Type":"ContainerStarted","Data":"c146936c47f481a0dfd5212eb022fb5f60745a9a1ca1393e3525a5d7554c5c30"}
Jul 08 19:31:11.415932 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:11.415893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" event={"ID":"1ecad761-77ff-4775-a354-a8bddcf22520","Type":"ContainerStarted","Data":"a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e"}
Jul 08 19:31:11.442185 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:11.442133    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" podStartSLOduration=2.442118683 podStartE2EDuration="2.442118683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:31:11.43979333 +0000 UTC m=+1930.083814060" watchObservedRunningTime="2026-07-08 19:31:11.442118683 +0000 UTC m=+1930.086139416"
Jul 08 19:31:12.229473 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:12.229438    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:12.234239 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:12.234216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:12.420955 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:12.420918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:12.421892 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:12.421874    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:13.425952 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.425907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:13.512283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.512251    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"]
Jul 08 19:31:13.515691 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.515667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.519338 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.519318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-71ba58d9-serving-cert\""
Jul 08 19:31:13.527662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.527600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"]
Jul 08 19:31:13.535740 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.535717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:13.659318 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.659285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299e18e-2ecc-4a89-9660-e1f068b7a431-openshift-service-ca\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.659909 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.659886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d299e18e-2ecc-4a89-9660-e1f068b7a431-maas-api-tls\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.660119 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.660104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bwj\" (UniqueName: \"kubernetes.io/projected/d299e18e-2ecc-4a89-9660-e1f068b7a431-kube-api-access-94bwj\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.706241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.706133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:13.761731 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.761695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bwj\" (UniqueName: \"kubernetes.io/projected/d299e18e-2ecc-4a89-9660-e1f068b7a431-kube-api-access-94bwj\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.761912 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.761784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299e18e-2ecc-4a89-9660-e1f068b7a431-openshift-service-ca\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.761912 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.761830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d299e18e-2ecc-4a89-9660-e1f068b7a431-maas-api-tls\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.762422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.762401    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299e18e-2ecc-4a89-9660-e1f068b7a431-openshift-service-ca\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.764228 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.764210    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d299e18e-2ecc-4a89-9660-e1f068b7a431-maas-api-tls\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.773483 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.773460    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bwj\" (UniqueName: \"kubernetes.io/projected/d299e18e-2ecc-4a89-9660-e1f068b7a431-kube-api-access-94bwj\") pod \"maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.828279 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.828255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:13.862758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.862727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:13.981803 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:13.981715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"]
Jul 08 19:31:13.986567 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:31:13.986538    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd299e18e_2ecc_4a89_9660_e1f068b7a431.slice/crio-2dea2e8d804e1f9a1376a81dffad260bfcf7fa1817f58f6a3361c5dad7cc7870 WatchSource:0}: Error finding container 2dea2e8d804e1f9a1376a81dffad260bfcf7fa1817f58f6a3361c5dad7cc7870: Status 404 returned error can't find the container with id 2dea2e8d804e1f9a1376a81dffad260bfcf7fa1817f58f6a3361c5dad7cc7870
Jul 08 19:31:14.362720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:14.362681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:14.439076 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:14.439034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv" event={"ID":"d299e18e-2ecc-4a89-9660-e1f068b7a431","Type":"ContainerStarted","Data":"2dea2e8d804e1f9a1376a81dffad260bfcf7fa1817f58f6a3361c5dad7cc7870"}
Jul 08 19:31:14.485038 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:14.485003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:15.446124 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:15.446093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv" event={"ID":"d299e18e-2ecc-4a89-9660-e1f068b7a431","Type":"ContainerStarted","Data":"c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f"}
Jul 08 19:31:15.446577 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:15.446347    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:17.979718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:17.979683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:18.674249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:18.674202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:18.786271 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:18.786230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:19.530788 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:19.530760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:21.462405 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:21.462365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:21.489289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:21.489235    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv" podStartSLOduration=7.851939106 podStartE2EDuration="8.489220627s" totalImagesPullingTime="637.281521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:31:15.467220436 +0000 UTC m=+1934.111241167" watchObservedRunningTime="2026-07-08 19:31:21.489220627 +0000 UTC m=+1940.133241356"
Jul 08 19:31:25.997085 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:25.997036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:26.092527 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:26.092489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:26.605786 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:26.605749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:27.134966 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.134928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:27.229959 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.229925    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"]
Jul 08 19:31:27.230173 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.230146    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv" podUID="d299e18e-2ecc-4a89-9660-e1f068b7a431" containerName="maas-api" containerID="cri-o://c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f" gracePeriod=30
Jul 08 19:31:27.281116 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.281082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:27.491911 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.491886    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:27.510964 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.510928    2563 generic.go:356] "Generic (PLEG): container finished" podID="d299e18e-2ecc-4a89-9660-e1f068b7a431" containerID="c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f" exitCode=0
Jul 08 19:31:27.511125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.510977    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"
Jul 08 19:31:27.511125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.511003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv" event={"ID":"d299e18e-2ecc-4a89-9660-e1f068b7a431","Type":"ContainerDied","Data":"c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f"}
Jul 08 19:31:27.511125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.511072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv" event={"ID":"d299e18e-2ecc-4a89-9660-e1f068b7a431","Type":"ContainerDied","Data":"2dea2e8d804e1f9a1376a81dffad260bfcf7fa1817f58f6a3361c5dad7cc7870"}
Jul 08 19:31:27.511125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.511096    2563 scope.go:117] "RemoveContainer" containerID="c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f"
Jul 08 19:31:27.519909 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.519887    2563 scope.go:117] "RemoveContainer" containerID="c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f"
Jul 08 19:31:27.520180 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:31:27.520152    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f\": container with ID starting with c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f not found: ID does not exist" containerID="c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f"
Jul 08 19:31:27.520243 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.520186    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f"} err="failed to get container status \"c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f\": rpc error: code = NotFound desc = could not find container \"c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f\": container with ID starting with c2a3bd8f778d276af3d44079a5af8f8fde7c96c86aaf24ed86da34cd5162f88f not found: ID does not exist"
Jul 08 19:31:27.593859 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.593828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299e18e-2ecc-4a89-9660-e1f068b7a431-openshift-service-ca\") pod \"d299e18e-2ecc-4a89-9660-e1f068b7a431\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") "
Jul 08 19:31:27.594035 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.593881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d299e18e-2ecc-4a89-9660-e1f068b7a431-maas-api-tls\") pod \"d299e18e-2ecc-4a89-9660-e1f068b7a431\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") "
Jul 08 19:31:27.594035 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.593963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94bwj\" (UniqueName: \"kubernetes.io/projected/d299e18e-2ecc-4a89-9660-e1f068b7a431-kube-api-access-94bwj\") pod \"d299e18e-2ecc-4a89-9660-e1f068b7a431\" (UID: \"d299e18e-2ecc-4a89-9660-e1f068b7a431\") "
Jul 08 19:31:27.594249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.594219    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d299e18e-2ecc-4a89-9660-e1f068b7a431-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d299e18e-2ecc-4a89-9660-e1f068b7a431" (UID: "d299e18e-2ecc-4a89-9660-e1f068b7a431"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:31:27.596023 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.595999    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d299e18e-2ecc-4a89-9660-e1f068b7a431-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d299e18e-2ecc-4a89-9660-e1f068b7a431" (UID: "d299e18e-2ecc-4a89-9660-e1f068b7a431"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:31:27.596117 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.596066    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d299e18e-2ecc-4a89-9660-e1f068b7a431-kube-api-access-94bwj" (OuterVolumeSpecName: "kube-api-access-94bwj") pod "d299e18e-2ecc-4a89-9660-e1f068b7a431" (UID: "d299e18e-2ecc-4a89-9660-e1f068b7a431"). InnerVolumeSpecName "kube-api-access-94bwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:31:27.695305 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.695236    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bwj\" (UniqueName: \"kubernetes.io/projected/d299e18e-2ecc-4a89-9660-e1f068b7a431-kube-api-access-94bwj\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:27.695305 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.695271    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d299e18e-2ecc-4a89-9660-e1f068b7a431-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:27.695305 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.695281    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d299e18e-2ecc-4a89-9660-e1f068b7a431-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:27.727658 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.727583    2563 patch_prober.go:28] interesting pod/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:34966->10.133.0.56:15021: read: connection reset by peer" start-of-body=
Jul 08 19:31:27.727794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.727700    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:34966->10.133.0.56:15021: read: connection reset by peer"
Jul 08 19:31:27.841036 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.841011    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"]
Jul 08 19:31:27.845350 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:27.845329    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71ba58d9-69d9d65b4f-wq6nv"]
Jul 08 19:31:28.455180 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:28.455148    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d299e18e-2ecc-4a89-9660-e1f068b7a431" path="/var/lib/kubelet/pods/d299e18e-2ecc-4a89-9660-e1f068b7a431/volumes"
Jul 08 19:31:28.518170 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:28.518094    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ecad761-77ff-4775-a354-a8bddcf22520" containerID="a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e" exitCode=0
Jul 08 19:31:28.518312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:28.518168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" event={"ID":"1ecad761-77ff-4775-a354-a8bddcf22520","Type":"ContainerDied","Data":"a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e"}
Jul 08 19:31:28.518577 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:28.518549    2563 scope.go:117] "RemoveContainer" containerID="a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e"
Jul 08 19:31:28.519562 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:28.519536    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:28.519656 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:28.519616    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:28.519705 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:28.519690    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:29.526856 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:29.526823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" event={"ID":"1ecad761-77ff-4775-a354-a8bddcf22520","Type":"ContainerStarted","Data":"00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd"}
Jul 08 19:31:30.206385 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.206354    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"
Jul 08 19:31:30.215836 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.215814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpbtg\" (UniqueName: \"kubernetes.io/projected/d13c1a0e-5124-488a-b079-b2c9c27459f7-kube-api-access-lpbtg\") pod \"d13c1a0e-5124-488a-b079-b2c9c27459f7\" (UID: \"d13c1a0e-5124-488a-b079-b2c9c27459f7\") "
Jul 08 19:31:30.217832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.217811    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13c1a0e-5124-488a-b079-b2c9c27459f7-kube-api-access-lpbtg" (OuterVolumeSpecName: "kube-api-access-lpbtg") pod "d13c1a0e-5124-488a-b079-b2c9c27459f7" (UID: "d13c1a0e-5124-488a-b079-b2c9c27459f7"). InnerVolumeSpecName "kube-api-access-lpbtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:31:30.229347 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.229314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:30.229532 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.229510    2563 patch_prober.go:28] interesting pod/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 08 19:31:30.229613 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.229593    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 08 19:31:30.317193 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.317110    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpbtg\" (UniqueName: \"kubernetes.io/projected/d13c1a0e-5124-488a-b079-b2c9c27459f7-kube-api-access-lpbtg\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:30.533199 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.533167    2563 generic.go:356] "Generic (PLEG): container finished" podID="d13c1a0e-5124-488a-b079-b2c9c27459f7" containerID="129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38" exitCode=137
Jul 08 19:31:30.533683 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.533214    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"
Jul 08 19:31:30.533683 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.533254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m" event={"ID":"d13c1a0e-5124-488a-b079-b2c9c27459f7","Type":"ContainerDied","Data":"129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38"}
Jul 08 19:31:30.533683 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.533313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m" event={"ID":"d13c1a0e-5124-488a-b079-b2c9c27459f7","Type":"ContainerDied","Data":"c25aba5785b6845c0d5ecd627a0b55ba5eb8d32ee72c1f31c2092d575abfc0d8"}
Jul 08 19:31:30.533683 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.533341    2563 scope.go:117] "RemoveContainer" containerID="129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38"
Jul 08 19:31:30.550168 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.550143    2563 scope.go:117] "RemoveContainer" containerID="129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38"
Jul 08 19:31:30.550539 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:31:30.550510    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38\": container with ID starting with 129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38 not found: ID does not exist" containerID="129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38"
Jul 08 19:31:30.550676 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.550549    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38"} err="failed to get container status \"129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38\": rpc error: code = NotFound desc = could not find container \"129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38\": container with ID starting with 129874997c84f69985a3d11ec06c802f5c3e5b95d9fb3d43805c2f134b842c38 not found: ID does not exist"
Jul 08 19:31:30.565132 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.565108    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"]
Jul 08 19:31:30.568432 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:30.568382    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-6gr6m"]
Jul 08 19:31:31.230052 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:31.230015    2563 patch_prober.go:28] interesting pod/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 08 19:31:31.230239 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:31.230075    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 08 19:31:32.230092 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:32.230054    2563 patch_prober.go:28] interesting pod/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 08 19:31:32.230480 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:32.230117    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 08 19:31:32.459507 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:32.459465    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d13c1a0e-5124-488a-b079-b2c9c27459f7" path="/var/lib/kubelet/pods/d13c1a0e-5124-488a-b079-b2c9c27459f7/volumes"
Jul 08 19:31:33.230041 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:33.230003    2563 patch_prober.go:28] interesting pod/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 08 19:31:33.230217 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:33.230061    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 08 19:31:33.746603 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:33.746557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:33.847486 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:33.847450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:34.034862 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:34.034780    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"]
Jul 08 19:31:34.035019 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:34.034963    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy" containerID="cri-o://00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd" gracePeriod=30
Jul 08 19:31:34.050111 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:34.050074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:34.435251 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:34.435153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:34.589507 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:34.589477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:34.822554 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:34.822513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:39.283563 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.283540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:39.407022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.406932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecad761-77ff-4775-a354-a8bddcf22520-istio-podinfo\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.406965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-data\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.406999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-socket\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-istio-token\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407119    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecad761-77ff-4775-a354-a8bddcf22520-istiod-ca-cert\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-certs\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407226    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-credential-socket\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407250    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:31:39.407372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-envoy\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407372 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407367    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g6pw\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-kube-api-access-2g6pw\") pod \"1ecad761-77ff-4775-a354-a8bddcf22520\" (UID: \"1ecad761-77ff-4775-a354-a8bddcf22520\") "
Jul 08 19:31:39.407747 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407428    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:31:39.407747 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407455    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:31:39.407747 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407492    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecad761-77ff-4775-a354-a8bddcf22520-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:31:39.407747 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407741    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.407957 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407760    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.407957 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407773    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecad761-77ff-4775-a354-a8bddcf22520-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.407957 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407785    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.407957 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.407938    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:31:39.409253 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.409232    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ecad761-77ff-4775-a354-a8bddcf22520-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:31:39.409429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.409410    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:31:39.409706 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.409684    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:31:39.409764 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.409725    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-kube-api-access-2g6pw" (OuterVolumeSpecName: "kube-api-access-2g6pw") pod "1ecad761-77ff-4775-a354-a8bddcf22520" (UID: "1ecad761-77ff-4775-a354-a8bddcf22520"). InnerVolumeSpecName "kube-api-access-2g6pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:31:39.508653 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.508597    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecad761-77ff-4775-a354-a8bddcf22520-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.508653 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.508649    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.508653 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.508662    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.508884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.508674    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecad761-77ff-4775-a354-a8bddcf22520-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.508884 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.508685    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g6pw\" (UniqueName: \"kubernetes.io/projected/1ecad761-77ff-4775-a354-a8bddcf22520-kube-api-access-2g6pw\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:31:39.591828 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.591797    2563 generic.go:356] "Generic (PLEG): container finished" podID="1ecad761-77ff-4775-a354-a8bddcf22520" containerID="00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd" exitCode=0
Jul 08 19:31:39.591828 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.591838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" event={"ID":"1ecad761-77ff-4775-a354-a8bddcf22520","Type":"ContainerDied","Data":"00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd"}
Jul 08 19:31:39.592029 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.591885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c" event={"ID":"1ecad761-77ff-4775-a354-a8bddcf22520","Type":"ContainerDied","Data":"c146936c47f481a0dfd5212eb022fb5f60745a9a1ca1393e3525a5d7554c5c30"}
Jul 08 19:31:39.592029 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.591907    2563 scope.go:117] "RemoveContainer" containerID="00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd"
Jul 08 19:31:39.592029 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.591842    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"
Jul 08 19:31:39.601135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.601118    2563 scope.go:117] "RemoveContainer" containerID="a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e"
Jul 08 19:31:39.608932 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.608913    2563 scope.go:117] "RemoveContainer" containerID="00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd"
Jul 08 19:31:39.609198 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:31:39.609177    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd\": container with ID starting with 00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd not found: ID does not exist" containerID="00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd"
Jul 08 19:31:39.609261 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.609205    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd"} err="failed to get container status \"00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd\": rpc error: code = NotFound desc = could not find container \"00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd\": container with ID starting with 00322da995429bd05f8a07211ec260dee311b779b7c96e350108e9b80ef26ffd not found: ID does not exist"
Jul 08 19:31:39.609261 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.609221    2563 scope.go:117] "RemoveContainer" containerID="a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e"
Jul 08 19:31:39.609466 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:31:39.609449    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e\": container with ID starting with a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e not found: ID does not exist" containerID="a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e"
Jul 08 19:31:39.609509 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.609473    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e"} err="failed to get container status \"a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e\": rpc error: code = NotFound desc = could not find container \"a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e\": container with ID starting with a5c2f5b5ce49204830874099df0601a44fa69fb5c4156029bdb8e9196c16068e not found: ID does not exist"
Jul 08 19:31:39.616392 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.616359    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"]
Jul 08 19:31:39.622201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:39.622180    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-71ba58d9-openshift-default-bfc47d7c5-scz7c"]
Jul 08 19:31:40.455716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:40.455686    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" path="/var/lib/kubelet/pods/1ecad761-77ff-4775-a354-a8bddcf22520/volumes"
Jul 08 19:31:41.319744 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:41.319712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:42.048341 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:42.048297    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:42.149085 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:42.149037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:48.595109 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:48.595065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:48.705989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:48.705957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:55.536244 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.536191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:55.833067 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.832992    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"]
Jul 08 19:31:55.833351 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833338    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d299e18e-2ecc-4a89-9660-e1f068b7a431" containerName="maas-api"
Jul 08 19:31:55.833403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833352    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d299e18e-2ecc-4a89-9660-e1f068b7a431" containerName="maas-api"
Jul 08 19:31:55.833403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833366    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy"
Jul 08 19:31:55.833403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy"
Jul 08 19:31:55.833403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833384    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13c1a0e-5124-488a-b079-b2c9c27459f7" containerName="cleanup"
Jul 08 19:31:55.833403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833390    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13c1a0e-5124-488a-b079-b2c9c27459f7" containerName="cleanup"
Jul 08 19:31:55.833556 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833449    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d299e18e-2ecc-4a89-9660-e1f068b7a431" containerName="maas-api"
Jul 08 19:31:55.833556 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833457    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy"
Jul 08 19:31:55.833556 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.833464    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13c1a0e-5124-488a-b079-b2c9c27459f7" containerName="cleanup"
Jul 08 19:31:55.836123 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.836106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.838928 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.838906    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-90938a12-openshift-default-dockercfg-j8ms7\""
Jul 08 19:31:55.852895 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.852869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"]
Jul 08 19:31:55.882153 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.882119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:55.956242 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvq4\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-kube-api-access-vsvq4\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956408 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-certs\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956408 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-envoy\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956408 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-socket\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956408 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istiod-ca-cert\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-credential-socket\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-podinfo\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-token\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:55.956662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:55.956540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-data\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057595 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-token\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057595 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-data\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvq4\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-kube-api-access-vsvq4\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-certs\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-envoy\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-socket\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istiod-ca-cert\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.057861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-credential-socket\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.058146 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.057877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-podinfo\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.058205 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.058135    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-data\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.058261 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.058206    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-certs\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.058337 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.058313    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-credential-socket\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.058418 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.058391    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-socket\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.058506 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.058489    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istiod-ca-cert\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.060181 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.060160    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-envoy\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.060281 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.060259    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-podinfo\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.068297 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.068264    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-token\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.068465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.068444    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvq4\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-kube-api-access-vsvq4\") pod \"e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") " pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.147039 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.146954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:56.288360 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.288333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"]
Jul 08 19:31:56.290863 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:31:56.290830    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79caeb91_bbe1_4301_a6f8_0caee3354ad2.slice/crio-bc664ea225425cecba715f68aefcf598758b04c638af0a26cd7f23ba7fe268bd WatchSource:0}: Error finding container bc664ea225425cecba715f68aefcf598758b04c638af0a26cd7f23ba7fe268bd: Status 404 returned error can't find the container with id bc664ea225425cecba715f68aefcf598758b04c638af0a26cd7f23ba7fe268bd
Jul 08 19:31:56.293557 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.293524    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:56.293671 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.293605    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:56.293721 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.293667    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:31:56.383818 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.383769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:56.681803 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.681764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" event={"ID":"79caeb91-bbe1-4301-a6f8-0caee3354ad2","Type":"ContainerStarted","Data":"18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b"}
Jul 08 19:31:56.682251 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.681850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" event={"ID":"79caeb91-bbe1-4301-a6f8-0caee3354ad2","Type":"ContainerStarted","Data":"bc664ea225425cecba715f68aefcf598758b04c638af0a26cd7f23ba7fe268bd"}
Jul 08 19:31:56.707087 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:56.707033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" podStartSLOduration=1.707018224 podStartE2EDuration="1.707018224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:31:56.704726554 +0000 UTC m=+1975.348747296" watchObservedRunningTime="2026-07-08 19:31:56.707018224 +0000 UTC m=+1975.351038952"
Jul 08 19:31:57.147697 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:57.147660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:57.152498 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:57.152471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:57.688193 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:57.688163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:57.689266 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:57.689247    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:31:58.662724 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.662677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:58.759716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.759679    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"]
Jul 08 19:31:58.760213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.760194    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy"
Jul 08 19:31:58.760290 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.760216    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy"
Jul 08 19:31:58.760345 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.760332    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecad761-77ff-4775-a354-a8bddcf22520" containerName="istio-proxy"
Jul 08 19:31:58.763545 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.763524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.767289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.767269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-90938a12-serving-cert\""
Jul 08 19:31:58.777448 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.777424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"]
Jul 08 19:31:58.779459 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.779429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:58.881124 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.881091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:58.885955 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.885928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlgwq\" (UniqueName: \"kubernetes.io/projected/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-kube-api-access-wlgwq\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.886096 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.885993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-maas-api-tls\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.886096 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.886011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-openshift-service-ca\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.986842 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.986752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-maas-api-tls\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.986842 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.986786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-openshift-service-ca\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.987062 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.986860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgwq\" (UniqueName: \"kubernetes.io/projected/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-kube-api-access-wlgwq\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.987586 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.987565    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-openshift-service-ca\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.988367 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.988338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:58.989509 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.989488    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-maas-api-tls\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:58.995354 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:58.995335    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlgwq\" (UniqueName: \"kubernetes.io/projected/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-kube-api-access-wlgwq\") pod \"maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:59.074846 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:59.074810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:31:59.216218 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:59.216191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"]
Jul 08 19:31:59.217605 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:31:59.217574    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3107c4e_2f60_4d19_9d0a_910167b3dd2b.slice/crio-bcb424a8347e962896b2f4a626c759832a4c08a05f2d522f06e8a1ce8918819f WatchSource:0}: Error finding container bcb424a8347e962896b2f4a626c759832a4c08a05f2d522f06e8a1ce8918819f: Status 404 returned error can't find the container with id bcb424a8347e962896b2f4a626c759832a4c08a05f2d522f06e8a1ce8918819f
Jul 08 19:31:59.381513 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:59.381479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:59.493635 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:59.493597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:31:59.700283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:31:59.700257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk" event={"ID":"f3107c4e-2f60-4d19-9d0a-910167b3dd2b","Type":"ContainerStarted","Data":"bcb424a8347e962896b2f4a626c759832a4c08a05f2d522f06e8a1ce8918819f"}
Jul 08 19:32:00.707249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:00.707211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk" event={"ID":"f3107c4e-2f60-4d19-9d0a-910167b3dd2b","Type":"ContainerStarted","Data":"b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507"}
Jul 08 19:32:00.707742 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:00.707286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:32:00.734455 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:00.734399    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk" podStartSLOduration=2.298042361 podStartE2EDuration="2.734385249s" totalImagesPullingTime="436.342888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:32:00.731824635 +0000 UTC m=+1979.375845361" watchObservedRunningTime="2026-07-08 19:32:00.734385249 +0000 UTC m=+1979.378405985"
Jul 08 19:32:01.974014 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:01.973985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:02.084665 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:02.084609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:03.168008 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:03.167942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:06.718554 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:06.718524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:32:09.859821 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:09.859783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:13.516140 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.515514    2563 patch_prober.go:28] interesting pod/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:39710->10.133.0.58:15021: read: connection reset by peer" start-of-body=
Jul 08 19:32:13.516140 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.515589    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:39710->10.133.0.58:15021: read: connection reset by peer"
Jul 08 19:32:13.787665 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.787545    2563 generic.go:356] "Generic (PLEG): container finished" podID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerID="18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b" exitCode=0
Jul 08 19:32:13.787665 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.787609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" event={"ID":"79caeb91-bbe1-4301-a6f8-0caee3354ad2","Type":"ContainerDied","Data":"18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b"}
Jul 08 19:32:13.788133 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.788114    2563 scope.go:117] "RemoveContainer" containerID="18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b"
Jul 08 19:32:13.789200 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.789169    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:32:13.789273 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.789244    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:32:13.789314 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:13.789274    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:32:14.794782 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:14.794742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" event={"ID":"79caeb91-bbe1-4301-a6f8-0caee3354ad2","Type":"ContainerStarted","Data":"821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5"}
Jul 08 19:32:15.147724 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:15.147610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:32:15.147980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:15.147949    2563 patch_prober.go:28] interesting pod/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 08 19:32:15.148166 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:15.148139    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 08 19:32:16.148420 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:16.148378    2563 patch_prober.go:28] interesting pod/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 08 19:32:16.148967 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:16.148455    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 08 19:32:17.085589 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:17.085548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:17.148175 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:17.148124    2563 patch_prober.go:28] interesting pod/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 08 19:32:17.148381 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:17.148195    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 08 19:32:17.204934 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:17.204898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:18.147972 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:18.147933    2563 patch_prober.go:28] interesting pod/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 08 19:32:18.148283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:18.148251    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 08 19:32:19.151609 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:19.151577    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:32:19.152014 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:19.151987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:32:19.152711 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:19.152691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:32:21.591174 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:21.591128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:21.850462 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:21.850375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:30.602598 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:30.602564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:30.973561 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:30.973475    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"]
Jul 08 19:32:30.973804 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:30.973763    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk" podUID="f3107c4e-2f60-4d19-9d0a-910167b3dd2b" containerName="maas-api" containerID="cri-o://b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507" gracePeriod=30
Jul 08 19:32:31.012271 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.012239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:31.224113 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.224054    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:32:31.296616 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.296586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-openshift-service-ca\") pod \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") "
Jul 08 19:32:31.296789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.296669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-maas-api-tls\") pod \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") "
Jul 08 19:32:31.296789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.296703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlgwq\" (UniqueName: \"kubernetes.io/projected/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-kube-api-access-wlgwq\") pod \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\" (UID: \"f3107c4e-2f60-4d19-9d0a-910167b3dd2b\") "
Jul 08 19:32:31.296996 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.296972    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3107c4e-2f60-4d19-9d0a-910167b3dd2b" (UID: "f3107c4e-2f60-4d19-9d0a-910167b3dd2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:32:31.297092 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.297072    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:32:31.298800 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.298772    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-kube-api-access-wlgwq" (OuterVolumeSpecName: "kube-api-access-wlgwq") pod "f3107c4e-2f60-4d19-9d0a-910167b3dd2b" (UID: "f3107c4e-2f60-4d19-9d0a-910167b3dd2b"). InnerVolumeSpecName "kube-api-access-wlgwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:32:31.298888 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.298815    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3107c4e-2f60-4d19-9d0a-910167b3dd2b" (UID: "f3107c4e-2f60-4d19-9d0a-910167b3dd2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:32:31.398143 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.398113    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:32:31.398143 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.398139    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlgwq\" (UniqueName: \"kubernetes.io/projected/f3107c4e-2f60-4d19-9d0a-910167b3dd2b-kube-api-access-wlgwq\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:32:31.883004 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.882970    2563 generic.go:356] "Generic (PLEG): container finished" podID="f3107c4e-2f60-4d19-9d0a-910167b3dd2b" containerID="b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507" exitCode=0
Jul 08 19:32:31.883422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.883018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk" event={"ID":"f3107c4e-2f60-4d19-9d0a-910167b3dd2b","Type":"ContainerDied","Data":"b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507"}
Jul 08 19:32:31.883422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.883055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk" event={"ID":"f3107c4e-2f60-4d19-9d0a-910167b3dd2b","Type":"ContainerDied","Data":"bcb424a8347e962896b2f4a626c759832a4c08a05f2d522f06e8a1ce8918819f"}
Jul 08 19:32:31.883422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.883076    2563 scope.go:117] "RemoveContainer" containerID="b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507"
Jul 08 19:32:31.883422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.883019    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"
Jul 08 19:32:31.892757 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.892738    2563 scope.go:117] "RemoveContainer" containerID="b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507"
Jul 08 19:32:31.893003 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:32:31.892980    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507\": container with ID starting with b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507 not found: ID does not exist" containerID="b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507"
Jul 08 19:32:31.893072 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.893017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507"} err="failed to get container status \"b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507\": rpc error: code = NotFound desc = could not find container \"b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507\": container with ID starting with b786a0864cc3be573f69d19346aaab0dd3e1fd49b4e89545519199beaf09e507 not found: ID does not exist"
Jul 08 19:32:31.912821 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.912791    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"]
Jul 08 19:32:31.920505 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:31.920484    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-90938a12-99b69f5dc-8f2fk"]
Jul 08 19:32:32.455818 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:32.455789    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3107c4e-2f60-4d19-9d0a-910167b3dd2b" path="/var/lib/kubelet/pods/f3107c4e-2f60-4d19-9d0a-910167b3dd2b/volumes"
Jul 08 19:32:36.701089 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:36.701050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:32:41.784247 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:32:41.784210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:17.934139 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:17.934101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:19.628819 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:19.628775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:24.908578 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:24.908492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:24.978164 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:24.978125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:25.167982 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:25.167895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"]
Jul 08 19:33:25.168156 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:25.168104    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy" containerID="cri-o://821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5" gracePeriod=30
Jul 08 19:33:25.186801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:25.186774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:26.526020 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:26.525982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:26.625235 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:26.625201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:26.726998 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:26.726958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:27.481102 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:27.481069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:27.579149 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:27.579116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:27.679143 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:27.679103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:29.062449 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:29.062399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:30.419827 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.419805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:33:30.546033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546001    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsvq4\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-kube-api-access-vsvq4\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546221 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-credential-socket\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546221 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546071    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-podinfo\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546221 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-data\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546221 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-envoy\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546435 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546286    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-certs\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546435 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546317    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:33:30.546435 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istiod-ca-cert\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546609 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-token\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546609 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546518    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-socket\") pod \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\" (UID: \"79caeb91-bbe1-4301-a6f8-0caee3354ad2\") "
Jul 08 19:33:30.546609 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546528    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:33:30.546805 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546739    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:33:30.546878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546861    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.546939 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546885    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.546939 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546900    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.547047 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.546955    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:33:30.547107 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.547078    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-data" (OuterVolumeSpecName: "istio-data") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:33:30.548368 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.548345    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:33:30.548368 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.548355    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-kube-api-access-vsvq4" (OuterVolumeSpecName: "kube-api-access-vsvq4") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "kube-api-access-vsvq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:33:30.548635 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.548604    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:33:30.548691 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.548670    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-token" (OuterVolumeSpecName: "istio-token") pod "79caeb91-bbe1-4301-a6f8-0caee3354ad2" (UID: "79caeb91-bbe1-4301-a6f8-0caee3354ad2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:33:30.647518 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.647476    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.647518 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.647514    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.647793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.647527    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsvq4\" (UniqueName: \"kubernetes.io/projected/79caeb91-bbe1-4301-a6f8-0caee3354ad2-kube-api-access-vsvq4\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.647793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.647544    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.647793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.647557    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:30.647793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:30.647569    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79caeb91-bbe1-4301-a6f8-0caee3354ad2-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:33:31.199053 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.199015    2563 generic.go:356] "Generic (PLEG): container finished" podID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerID="821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5" exitCode=0
Jul 08 19:33:31.199256 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.199075    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"
Jul 08 19:33:31.199256 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.199102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" event={"ID":"79caeb91-bbe1-4301-a6f8-0caee3354ad2","Type":"ContainerDied","Data":"821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5"}
Jul 08 19:33:31.199256 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.199171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs" event={"ID":"79caeb91-bbe1-4301-a6f8-0caee3354ad2","Type":"ContainerDied","Data":"bc664ea225425cecba715f68aefcf598758b04c638af0a26cd7f23ba7fe268bd"}
Jul 08 19:33:31.199256 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.199202    2563 scope.go:117] "RemoveContainer" containerID="821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5"
Jul 08 19:33:31.208575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.208557    2563 scope.go:117] "RemoveContainer" containerID="18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b"
Jul 08 19:33:31.216429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.216414    2563 scope.go:117] "RemoveContainer" containerID="821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5"
Jul 08 19:33:31.216695 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:33:31.216666    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5\": container with ID starting with 821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5 not found: ID does not exist" containerID="821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5"
Jul 08 19:33:31.216783 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.216705    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5"} err="failed to get container status \"821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5\": rpc error: code = NotFound desc = could not find container \"821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5\": container with ID starting with 821c1d904a9ab8bf07305633ede1ba5933cc4ff4d0fd6ee9d2050209630b5be5 not found: ID does not exist"
Jul 08 19:33:31.216783 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.216724    2563 scope.go:117] "RemoveContainer" containerID="18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b"
Jul 08 19:33:31.216917 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:33:31.216901    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b\": container with ID starting with 18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b not found: ID does not exist" containerID="18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b"
Jul 08 19:33:31.216958 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.216923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b"} err="failed to get container status \"18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b\": rpc error: code = NotFound desc = could not find container \"18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b\": container with ID starting with 18639fe75c0785a434b9d0fa54025c2512802958361021bb9db884f2d632df3b not found: ID does not exist"
Jul 08 19:33:31.224612 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.224591    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"]
Jul 08 19:33:31.228755 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:31.228734    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-90938a12-openshift-default-7f64d67f99-bnvfs"]
Jul 08 19:33:32.456331 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:32.456303    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" path="/var/lib/kubelet/pods/79caeb91-bbe1-4301-a6f8-0caee3354ad2/volumes"
Jul 08 19:33:35.099683 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:35.099640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:35.182004 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:35.181962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:35.793039 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:35.793001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:42.174237 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:42.174195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:42.276062 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:42.276030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:42.374365 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:42.374330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:52.930221 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:52.930185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:33:53.274862 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:33:53.274816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:02.380456 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:02.380427    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:34:02.387601 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:02.387578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:34:02.396676 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:02.396652    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:34:02.403201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:02.403183    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:34:14.260574 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:14.260535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:20.558269 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:20.558225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:20.625664 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:20.625615    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:20.818793 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:20.818706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:21.109735 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.109647    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"]
Jul 08 19:34:21.110040 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110026    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3107c4e-2f60-4d19-9d0a-910167b3dd2b" containerName="maas-api"
Jul 08 19:34:21.110040 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110041    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3107c4e-2f60-4d19-9d0a-910167b3dd2b" containerName="maas-api"
Jul 08 19:34:21.110135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110049    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy"
Jul 08 19:34:21.110135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy"
Jul 08 19:34:21.110135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy"
Jul 08 19:34:21.110135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy"
Jul 08 19:34:21.110135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110122    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy"
Jul 08 19:34:21.110135 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.110131    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3107c4e-2f60-4d19-9d0a-910167b3dd2b" containerName="maas-api"
Jul 08 19:34:21.113065 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.113050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.116307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.116285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5fb4a8-openshift-default-dockercfg-b6bwl\""
Jul 08 19:34:21.125253 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.125228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"]
Jul 08 19:34:21.141342 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.141305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:21.205004 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.204968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-envoy\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205179 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-token\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205179 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-credential-socket\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205179 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-socket\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205313 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-certs\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205313 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-podinfo\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205313 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istiod-ca-cert\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205516 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v589c\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-kube-api-access-v589c\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.205600 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.205579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-data\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-data\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-envoy\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-token\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-credential-socket\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-socket\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-certs\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-podinfo\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istiod-ca-cert\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v589c\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-kube-api-access-v589c\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306797    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-data\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306855    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-credential-socket\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.306971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.306928    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-socket\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.307249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.307224    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-certs\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.307369 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.307351    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istiod-ca-cert\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.308807 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.308777    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-envoy\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.309050 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.309034    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-podinfo\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.315402 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.315367    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-token\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.315603 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.315586    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v589c\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-kube-api-access-v589c\") pod \"e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") " pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.402640 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.402530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:21.424635 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.424604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:21.565046 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.565016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"]
Jul 08 19:34:21.566735 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:34:21.566710    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2296ae80_ddf8_420f_9f7c_d14e00fd9217.slice/crio-ae0abf8e5fbdebd3f4084d52cef53df486331af60c592b1b9463ab885df7e406 WatchSource:0}: Error finding container ae0abf8e5fbdebd3f4084d52cef53df486331af60c592b1b9463ab885df7e406: Status 404 returned error can't find the container with id ae0abf8e5fbdebd3f4084d52cef53df486331af60c592b1b9463ab885df7e406
Jul 08 19:34:21.569304 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.569270    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:21.569380 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.569333    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:21.569380 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:21.569359    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:22.458289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:22.458260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" event={"ID":"2296ae80-ddf8-420f-9f7c-d14e00fd9217","Type":"ContainerStarted","Data":"50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3"}
Jul 08 19:34:22.458440 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:22.458307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" event={"ID":"2296ae80-ddf8-420f-9f7c-d14e00fd9217","Type":"ContainerStarted","Data":"ae0abf8e5fbdebd3f4084d52cef53df486331af60c592b1b9463ab885df7e406"}
Jul 08 19:34:22.481729 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:22.481652    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" podStartSLOduration=1.481611369 podStartE2EDuration="1.481611369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:34:22.480244069 +0000 UTC m=+2121.124264843" watchObservedRunningTime="2026-07-08 19:34:22.481611369 +0000 UTC m=+2121.125632159"
Jul 08 19:34:23.425019 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:23.424982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:23.429882 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:23.429856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:23.463932 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:23.463906    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:23.464833 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:23.464817    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:24.429950 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.429917    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:24.531893 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.531863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:24.603289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.603261    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"]
Jul 08 19:34:24.603716 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.603699    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="79caeb91-bbe1-4301-a6f8-0caee3354ad2" containerName="istio-proxy"
Jul 08 19:34:24.606740 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.606720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.609646 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.609599    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5fb4a8-serving-cert\""
Jul 08 19:34:24.619200 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.619172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"]
Jul 08 19:34:24.637259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.637231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bpsr\" (UniqueName: \"kubernetes.io/projected/4726f4a5-fdf2-423c-8c65-062fcd35793d-kube-api-access-8bpsr\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.637373 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.637281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.637434 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.637372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4726f4a5-fdf2-423c-8c65-062fcd35793d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.643398 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.643376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:24.738707 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.738608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4726f4a5-fdf2-423c-8c65-062fcd35793d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.738878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.738727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bpsr\" (UniqueName: \"kubernetes.io/projected/4726f4a5-fdf2-423c-8c65-062fcd35793d-kube-api-access-8bpsr\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.738878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.738768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.738878 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:34:24.738865    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-serving-cert: secret "maas-api-e2e-inf-a-5fb4a8-serving-cert" not found
Jul 08 19:34:24.739056 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:34:24.738918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls podName:4726f4a5-fdf2-423c-8c65-062fcd35793d nodeName:}" failed. No retries permitted until 2026-07-08 19:34:25.238902139 +0000 UTC m=+2123.882922848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls") pod "maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" (UID: "4726f4a5-fdf2-423c-8c65-062fcd35793d") : secret "maas-api-e2e-inf-a-5fb4a8-serving-cert" not found
Jul 08 19:34:24.739370 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.739345    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4726f4a5-fdf2-423c-8c65-062fcd35793d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.754126 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.754101    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bpsr\" (UniqueName: \"kubernetes.io/projected/4726f4a5-fdf2-423c-8c65-062fcd35793d-kube-api-access-8bpsr\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:24.831689 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:24.831656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:25.177073 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:25.176964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:25.243973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:25.243938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:25.246393 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:25.246364    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls\") pod \"maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:25.277792 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:25.277763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:25.520666 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:25.520611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:25.662437 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:25.662409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"]
Jul 08 19:34:25.663564 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:34:25.663534    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4726f4a5_fdf2_423c_8c65_062fcd35793d.slice/crio-fe0e683a61d7f6fb3e593efa6075bb01b0ba57ef4094fb38ee2a9f60a623fc18 WatchSource:0}: Error finding container fe0e683a61d7f6fb3e593efa6075bb01b0ba57ef4094fb38ee2a9f60a623fc18: Status 404 returned error can't find the container with id fe0e683a61d7f6fb3e593efa6075bb01b0ba57ef4094fb38ee2a9f60a623fc18
Jul 08 19:34:25.665064 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:25.665043    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 19:34:26.479045 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:26.478953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" event={"ID":"4726f4a5-fdf2-423c-8c65-062fcd35793d","Type":"ContainerStarted","Data":"4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc"}
Jul 08 19:34:26.479045 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:26.479017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" event={"ID":"4726f4a5-fdf2-423c-8c65-062fcd35793d","Type":"ContainerStarted","Data":"fe0e683a61d7f6fb3e593efa6075bb01b0ba57ef4094fb38ee2a9f60a623fc18"}
Jul 08 19:34:26.479229 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:26.479057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:26.498888 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:26.498840    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" podStartSLOduration=1.95632173 podStartE2EDuration="2.498822167s" totalImagesPullingTime="542.500437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:34:26.498284577 +0000 UTC m=+2125.142305310" watchObservedRunningTime="2026-07-08 19:34:26.498822167 +0000 UTC m=+2125.142842898"
Jul 08 19:34:29.005149 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.005114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:29.368577 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.368483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:29.644736 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.644615    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"]
Jul 08 19:34:29.648315 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.648292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.651475 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.651455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fb0a08-openshift-default-dockercfg-dk6fd\""
Jul 08 19:34:29.660479 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.660457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"]
Jul 08 19:34:29.676979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.676953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:29.681822 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.681795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-envoy\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.681952 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.681837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-certs\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.681952 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.681912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-podinfo\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.682078 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.682011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-credential-socket\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.682078 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.682048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-token\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.682176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.682099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjn4\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-kube-api-access-wkjn4\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.682176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.682137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695d2159-f1e6-400d-9f5a-5180f68d7a14-istiod-ca-cert\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.682176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.682169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-data\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.682314 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.682249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-socket\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783056 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-envoy\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783056 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-certs\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-podinfo\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-credential-socket\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-token\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjn4\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-kube-api-access-wkjn4\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783515 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695d2159-f1e6-400d-9f5a-5180f68d7a14-istiod-ca-cert\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783515 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-data\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783515 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-socket\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783524    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-credential-socket\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783573    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-certs\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.783816 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783800    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-socket\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.784006 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.783984    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-data\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.784144 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.784125    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695d2159-f1e6-400d-9f5a-5180f68d7a14-istiod-ca-cert\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.785667 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.785647    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-envoy\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.785794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.785775    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-podinfo\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.793836 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.793815    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-token\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.794237 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.794216    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjn4\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-kube-api-access-wkjn4\") pod \"e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") " pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:29.959921 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:29.959828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:30.105402 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.105373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"]
Jul 08 19:34:30.106268 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:34:30.106244    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695d2159_f1e6_400d_9f5a_5180f68d7a14.slice/crio-31ec99533321abaab306e68cc51421ca9d4d9ba1d77aacc93c209f89924d2626 WatchSource:0}: Error finding container 31ec99533321abaab306e68cc51421ca9d4d9ba1d77aacc93c209f89924d2626: Status 404 returned error can't find the container with id 31ec99533321abaab306e68cc51421ca9d4d9ba1d77aacc93c209f89924d2626
Jul 08 19:34:30.108534 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.108505    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:30.108611 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.108565    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:30.108611 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.108595    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:30.252163 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.252127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:30.502575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.502483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" event={"ID":"695d2159-f1e6-400d-9f5a-5180f68d7a14","Type":"ContainerStarted","Data":"1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244"}
Jul 08 19:34:30.502575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.502538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" event={"ID":"695d2159-f1e6-400d-9f5a-5180f68d7a14","Type":"ContainerStarted","Data":"31ec99533321abaab306e68cc51421ca9d4d9ba1d77aacc93c209f89924d2626"}
Jul 08 19:34:30.527874 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.527826    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podStartSLOduration=1.527810704 podStartE2EDuration="1.527810704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:34:30.525718978 +0000 UTC m=+2129.169739710" watchObservedRunningTime="2026-07-08 19:34:30.527810704 +0000 UTC m=+2129.171831433"
Jul 08 19:34:30.960708 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:30.960606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:31.966173 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:31.966142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:32.491938 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.491908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:34:32.494038 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.494014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:32.514030 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.513993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:32.514732 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.514711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:32.607999 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.607963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:32.644837 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.644807    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"]
Jul 08 19:34:32.648916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.648891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.655897 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.655830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fb0a08-serving-cert\""
Jul 08 19:34:32.664989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.664960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"]
Jul 08 19:34:32.712570 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.712540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgcj\" (UniqueName: \"kubernetes.io/projected/85f6a421-d50f-4c79-a266-6935e1e6f4b5-kube-api-access-csgcj\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.712755 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.712605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.712855 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.712788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f6a421-d50f-4c79-a266-6935e1e6f4b5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.713830 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.713786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:32.813980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.813871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgcj\" (UniqueName: \"kubernetes.io/projected/85f6a421-d50f-4c79-a266-6935e1e6f4b5-kube-api-access-csgcj\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.813980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.813943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.814212 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.814049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f6a421-d50f-4c79-a266-6935e1e6f4b5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.814212 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:34:32.814114    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-serving-cert: secret "maas-api-e2e-inf-b-fb0a08-serving-cert" not found
Jul 08 19:34:32.814212 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:34:32.814203    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls podName:85f6a421-d50f-4c79-a266-6935e1e6f4b5 nodeName:}" failed. No retries permitted until 2026-07-08 19:34:33.314178143 +0000 UTC m=+2131.958198851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls") pod "maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb" (UID: "85f6a421-d50f-4c79-a266-6935e1e6f4b5") : secret "maas-api-e2e-inf-b-fb0a08-serving-cert" not found
Jul 08 19:34:32.814781 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.814759    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f6a421-d50f-4c79-a266-6935e1e6f4b5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:32.823652 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:32.823596    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgcj\" (UniqueName: \"kubernetes.io/projected/85f6a421-d50f-4c79-a266-6935e1e6f4b5-kube-api-access-csgcj\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:33.171313 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:33.171213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:33.268824 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:33.268766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:33.319311 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:33.319275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:33.322055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:33.322031    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:33.370779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:33.370735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:33.560188 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:33.560141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:33.725703 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:33.725674    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"]
Jul 08 19:34:33.726281 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:34:33.726257    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f6a421_d50f_4c79_a266_6935e1e6f4b5.slice/crio-e9d9c16591a1ac32655366759758e9754eed0258058c6dbe781d62dececf6048 WatchSource:0}: Error finding container e9d9c16591a1ac32655366759758e9754eed0258058c6dbe781d62dececf6048: Status 404 returned error can't find the container with id e9d9c16591a1ac32655366759758e9754eed0258058c6dbe781d62dececf6048
Jul 08 19:34:34.525349 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:34.525309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb" event={"ID":"85f6a421-d50f-4c79-a266-6935e1e6f4b5","Type":"ContainerStarted","Data":"53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346"}
Jul 08 19:34:34.525765 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:34.525371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb" event={"ID":"85f6a421-d50f-4c79-a266-6935e1e6f4b5","Type":"ContainerStarted","Data":"e9d9c16591a1ac32655366759758e9754eed0258058c6dbe781d62dececf6048"}
Jul 08 19:34:34.525765 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:34.525401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:34.558615 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:34.558553    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb" podStartSLOduration=2.122004628 podStartE2EDuration="2.558528613s" totalImagesPullingTime="436.523985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:34:34.55148947 +0000 UTC m=+2133.195510195" watchObservedRunningTime="2026-07-08 19:34:34.558528613 +0000 UTC m=+2133.202549344"
Jul 08 19:34:36.978025 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:36.977985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:39.505785 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:39.505744    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 19:34:39.506183 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:39.505813    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 19:34:40.535974 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:40.535946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:34:40.565654 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:40.565594    2563 generic.go:356] "Generic (PLEG): container finished" podID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerID="50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3" exitCode=0
Jul 08 19:34:40.565845 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:40.565746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" event={"ID":"2296ae80-ddf8-420f-9f7c-d14e00fd9217","Type":"ContainerDied","Data":"50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3"}
Jul 08 19:34:40.566234 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:40.566213    2563 scope.go:117] "RemoveContainer" containerID="50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3"
Jul 08 19:34:40.567416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:40.567377    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:40.567493 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:40.567476    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:40.567535 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:40.567523    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:41.576532 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:41.576486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" event={"ID":"2296ae80-ddf8-420f-9f7c-d14e00fd9217","Type":"ContainerStarted","Data":"91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd"}
Jul 08 19:34:42.425340 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:42.425303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:42.425542 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:42.425487    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 08 19:34:42.425542 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:42.425529    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 08 19:34:43.425498 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:43.425458    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 08 19:34:43.425929 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:43.425517    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 08 19:34:44.022834 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:44.022778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:44.425714 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:44.425601    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 08 19:34:44.425714 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:44.425687    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 08 19:34:45.426068 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:45.426022    2563 patch_prober.go:28] interesting pod/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 08 19:34:45.426426 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:45.426100    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 08 19:34:46.009194 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:46.009159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:46.131983 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:46.131948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:46.428751 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:46.428666    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:46.429206 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:46.429075    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:46.429828 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:46.429808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:34:47.515361 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.515328    2563 patch_prober.go:28] interesting pod/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 08 19:34:47.515786 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.515391    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 08 19:34:47.621330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.621298    2563 generic.go:356] "Generic (PLEG): container finished" podID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerID="1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244" exitCode=0
Jul 08 19:34:47.621507 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.621373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" event={"ID":"695d2159-f1e6-400d-9f5a-5180f68d7a14","Type":"ContainerDied","Data":"1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244"}
Jul 08 19:34:47.621922 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.621903    2563 scope.go:117] "RemoveContainer" containerID="1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244"
Jul 08 19:34:47.622913 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.622884    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:47.622983 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.622965    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:47.623033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:47.623019    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:34:48.629184 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:48.629150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" event={"ID":"695d2159-f1e6-400d-9f5a-5180f68d7a14","Type":"ContainerStarted","Data":"a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644"}
Jul 08 19:34:48.960935 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:48.960853    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:48.961232 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:48.961204    2563 patch_prober.go:28] interesting pod/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 08 19:34:48.961307 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:48.961279    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 08 19:34:49.960325 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:49.960286    2563 patch_prober.go:28] interesting pod/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 08 19:34:49.960953 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:49.960923    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 08 19:34:50.960381 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:50.960339    2563 patch_prober.go:28] interesting pod/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 08 19:34:50.960887 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:50.960409    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 08 19:34:51.960671 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:51.960507    2563 patch_prober.go:28] interesting pod/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 08 19:34:51.960671 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:51.960591    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 08 19:34:52.961265 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:52.961218    2563 patch_prober.go:28] interesting pod/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 08 19:34:52.961937 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:52.961288    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 08 19:34:53.964291 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:53.964258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:53.964805 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:53.964662    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:53.965403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:53.965383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:34:55.048550 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:55.048510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:34:55.149104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:34:55.149048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:02.488464 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:02.488433    2563 scope.go:117] "RemoveContainer" containerID="d3c481efb9e37daff7607b62c2700db29b996e60a3c0a4ef6e6304b3d21089a1"
Jul 08 19:35:17.489500 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:17.489464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:35.471540 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:35.471501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:36.401879 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:36.401837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:36.563171 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:36.563137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:36.979583 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:36.979543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:37.080551 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.080504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:37.181084 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.181039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:37.258597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.258557    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"]
Jul 08 19:35:37.258854 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.258824    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" podUID="4726f4a5-fdf2-423c-8c65-062fcd35793d" containerName="maas-api" containerID="cri-o://4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc" gracePeriod=30
Jul 08 19:35:37.289163 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.289137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:35:37.500265 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.500243    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:35:37.564743 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.564663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bpsr\" (UniqueName: \"kubernetes.io/projected/4726f4a5-fdf2-423c-8c65-062fcd35793d-kube-api-access-8bpsr\") pod \"4726f4a5-fdf2-423c-8c65-062fcd35793d\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") "
Jul 08 19:35:37.564743 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.564709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4726f4a5-fdf2-423c-8c65-062fcd35793d-openshift-service-ca\") pod \"4726f4a5-fdf2-423c-8c65-062fcd35793d\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") "
Jul 08 19:35:37.564743 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.564734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls\") pod \"4726f4a5-fdf2-423c-8c65-062fcd35793d\" (UID: \"4726f4a5-fdf2-423c-8c65-062fcd35793d\") "
Jul 08 19:35:37.565188 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.565042    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4726f4a5-fdf2-423c-8c65-062fcd35793d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4726f4a5-fdf2-423c-8c65-062fcd35793d" (UID: "4726f4a5-fdf2-423c-8c65-062fcd35793d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:35:37.565188 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.565162    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4726f4a5-fdf2-423c-8c65-062fcd35793d-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:35:37.566883 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.566861    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4726f4a5-fdf2-423c-8c65-062fcd35793d-kube-api-access-8bpsr" (OuterVolumeSpecName: "kube-api-access-8bpsr") pod "4726f4a5-fdf2-423c-8c65-062fcd35793d" (UID: "4726f4a5-fdf2-423c-8c65-062fcd35793d"). InnerVolumeSpecName "kube-api-access-8bpsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:35:37.566973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.566907    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4726f4a5-fdf2-423c-8c65-062fcd35793d" (UID: "4726f4a5-fdf2-423c-8c65-062fcd35793d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:35:37.666282 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.666250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bpsr\" (UniqueName: \"kubernetes.io/projected/4726f4a5-fdf2-423c-8c65-062fcd35793d-kube-api-access-8bpsr\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:35:37.666282 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.666278    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4726f4a5-fdf2-423c-8c65-062fcd35793d-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:35:37.894099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.894009    2563 generic.go:356] "Generic (PLEG): container finished" podID="4726f4a5-fdf2-423c-8c65-062fcd35793d" containerID="4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc" exitCode=0
Jul 08 19:35:37.894099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.894056    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"
Jul 08 19:35:37.894099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.894077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" event={"ID":"4726f4a5-fdf2-423c-8c65-062fcd35793d","Type":"ContainerDied","Data":"4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc"}
Jul 08 19:35:37.894403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.894141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" event={"ID":"4726f4a5-fdf2-423c-8c65-062fcd35793d","Type":"ContainerDied","Data":"fe0e683a61d7f6fb3e593efa6075bb01b0ba57ef4094fb38ee2a9f60a623fc18"}
Jul 08 19:35:37.894403 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.894172    2563 scope.go:117] "RemoveContainer" containerID="4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc"
Jul 08 19:35:37.906005 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.905980    2563 scope.go:117] "RemoveContainer" containerID="4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc"
Jul 08 19:35:37.906444 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:35:37.906423    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc\": container with ID starting with 4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc not found: ID does not exist" containerID="4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc"
Jul 08 19:35:37.906523 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.906455    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc"} err="failed to get container status \"4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc\": rpc error: code = NotFound desc = could not find container \"4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc\": container with ID starting with 4e489bdcaeba9121ddc2edd9eb8622874df7b3d1c8ac0751c0e5586bacc1bcfc not found: ID does not exist"
Jul 08 19:35:37.921811 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.921787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"]
Jul 08 19:35:37.925489 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:37.925458    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr"]
Jul 08 19:35:38.455644 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:38.455595    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4726f4a5-fdf2-423c-8c65-062fcd35793d" path="/var/lib/kubelet/pods/4726f4a5-fdf2-423c-8c65-062fcd35793d/volumes"
Jul 08 19:35:40.485958 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:40.485918    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5fb4a8-6c7b757d44-9cqwr" podUID="4726f4a5-fdf2-423c-8c65-062fcd35793d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": context deadline exceeded"
Jul 08 19:35:42.041754 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:35:42.041717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:19.161533 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:19.161489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:19.474894 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:19.474809    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"]
Jul 08 19:36:19.475092 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:19.475006    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy" containerID="cri-o://91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd" gracePeriod=30
Jul 08 19:36:19.476854 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:19.476825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:19.912404 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:19.912367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:20.363732 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:20.363697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:20.463432 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:20.463393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:20.565890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:20.565856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:20.641028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:20.640948    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"]
Jul 08 19:36:20.641257 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:20.641232    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb" podUID="85f6a421-d50f-4c79-a266-6935e1e6f4b5" containerName="maas-api" containerID="cri-o://53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346" gracePeriod=30
Jul 08 19:36:20.666302 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:20.666276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:21.090235 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.090211    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:36:21.121554 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.121528    2563 generic.go:356] "Generic (PLEG): container finished" podID="85f6a421-d50f-4c79-a266-6935e1e6f4b5" containerID="53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346" exitCode=0
Jul 08 19:36:21.121718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.121582    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"
Jul 08 19:36:21.121718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.121611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb" event={"ID":"85f6a421-d50f-4c79-a266-6935e1e6f4b5","Type":"ContainerDied","Data":"53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346"}
Jul 08 19:36:21.121718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.121690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb" event={"ID":"85f6a421-d50f-4c79-a266-6935e1e6f4b5","Type":"ContainerDied","Data":"e9d9c16591a1ac32655366759758e9754eed0258058c6dbe781d62dececf6048"}
Jul 08 19:36:21.121718 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.121713    2563 scope.go:117] "RemoveContainer" containerID="53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346"
Jul 08 19:36:21.130603 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.130577    2563 scope.go:117] "RemoveContainer" containerID="53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346"
Jul 08 19:36:21.130984 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:21.130955    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346\": container with ID starting with 53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346 not found: ID does not exist" containerID="53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346"
Jul 08 19:36:21.131092 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.130994    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346"} err="failed to get container status \"53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346\": rpc error: code = NotFound desc = could not find container \"53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346\": container with ID starting with 53c904a007bccec6fe39485d933535137f34a6d8d99aaeaf93f3868de7217346 not found: ID does not exist"
Jul 08 19:36:21.183357 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.183287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls\") pod \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") "
Jul 08 19:36:21.183357 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.183321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csgcj\" (UniqueName: \"kubernetes.io/projected/85f6a421-d50f-4c79-a266-6935e1e6f4b5-kube-api-access-csgcj\") pod \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") "
Jul 08 19:36:21.183530 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.183432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f6a421-d50f-4c79-a266-6935e1e6f4b5-openshift-service-ca\") pod \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\" (UID: \"85f6a421-d50f-4c79-a266-6935e1e6f4b5\") "
Jul 08 19:36:21.183803 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.183779    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f6a421-d50f-4c79-a266-6935e1e6f4b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85f6a421-d50f-4c79-a266-6935e1e6f4b5" (UID: "85f6a421-d50f-4c79-a266-6935e1e6f4b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:36:21.185295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.185274    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f6a421-d50f-4c79-a266-6935e1e6f4b5-kube-api-access-csgcj" (OuterVolumeSpecName: "kube-api-access-csgcj") pod "85f6a421-d50f-4c79-a266-6935e1e6f4b5" (UID: "85f6a421-d50f-4c79-a266-6935e1e6f4b5"). InnerVolumeSpecName "kube-api-access-csgcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:36:21.185384 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.185321    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85f6a421-d50f-4c79-a266-6935e1e6f4b5" (UID: "85f6a421-d50f-4c79-a266-6935e1e6f4b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:36:21.284899 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.284870    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f6a421-d50f-4c79-a266-6935e1e6f4b5-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:21.284899 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.284896    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f6a421-d50f-4c79-a266-6935e1e6f4b5-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:21.284899 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.284905    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csgcj\" (UniqueName: \"kubernetes.io/projected/85f6a421-d50f-4c79-a266-6935e1e6f4b5-kube-api-access-csgcj\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:21.444939 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.444872    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"]
Jul 08 19:36:21.450600 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:21.450575    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb0a08-66c586b4d4-56cmb"]
Jul 08 19:36:22.461210 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:22.461176    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f6a421-d50f-4c79-a266-6935e1e6f4b5" path="/var/lib/kubelet/pods/85f6a421-d50f-4c79-a266-6935e1e6f4b5/volumes"
Jul 08 19:36:24.726964 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.726939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:36:24.818310 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-credential-socket\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818470 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818319    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-certs\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818470 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-token\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818470 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istiod-ca-cert\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818470 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818397    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-envoy\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818470 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-podinfo\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818474    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v589c\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-kube-api-access-v589c\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818507    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-data\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-socket\") pod \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\" (UID: \"2296ae80-ddf8-420f-9f7c-d14e00fd9217\") "
Jul 08 19:36:24.818772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818602    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:24.818772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818617    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:24.819032 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818896    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:36:24.819032 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818953    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.819032 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.818974    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.819185 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.819037    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:24.819609 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.819582    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-data" (OuterVolumeSpecName: "istio-data") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:24.820616 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.820581    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-kube-api-access-v589c" (OuterVolumeSpecName: "kube-api-access-v589c") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "kube-api-access-v589c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:36:24.820616 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.820585    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:24.820756 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.820677    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:36:24.820756 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.820740    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-token" (OuterVolumeSpecName: "istio-token") pod "2296ae80-ddf8-420f-9f7c-d14e00fd9217" (UID: "2296ae80-ddf8-420f-9f7c-d14e00fd9217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:36:24.919560 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.919535    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.919560 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.919557    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.919730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.919565    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v589c\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-kube-api-access-v589c\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.919730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.919575    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.919730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.919584    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2296ae80-ddf8-420f-9f7c-d14e00fd9217-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.919730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.919592    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:24.919730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:24.919600    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2296ae80-ddf8-420f-9f7c-d14e00fd9217-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:25.145069 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.144980    2563 generic.go:356] "Generic (PLEG): container finished" podID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerID="91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd" exitCode=0
Jul 08 19:36:25.145069 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.145026    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"
Jul 08 19:36:25.145241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.145063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" event={"ID":"2296ae80-ddf8-420f-9f7c-d14e00fd9217","Type":"ContainerDied","Data":"91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd"}
Jul 08 19:36:25.145241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.145121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5" event={"ID":"2296ae80-ddf8-420f-9f7c-d14e00fd9217","Type":"ContainerDied","Data":"ae0abf8e5fbdebd3f4084d52cef53df486331af60c592b1b9463ab885df7e406"}
Jul 08 19:36:25.145241 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.145153    2563 scope.go:117] "RemoveContainer" containerID="91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd"
Jul 08 19:36:25.154804 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.154785    2563 scope.go:117] "RemoveContainer" containerID="50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3"
Jul 08 19:36:25.162742 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.162721    2563 scope.go:117] "RemoveContainer" containerID="91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd"
Jul 08 19:36:25.162976 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:25.162957    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd\": container with ID starting with 91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd not found: ID does not exist" containerID="91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd"
Jul 08 19:36:25.163025 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.162985    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd"} err="failed to get container status \"91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd\": rpc error: code = NotFound desc = could not find container \"91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd\": container with ID starting with 91cf8a46ccc330516d7a192bef0e76978aafa6348ac38c82d46009bd1730b0cd not found: ID does not exist"
Jul 08 19:36:25.163025 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.163001    2563 scope.go:117] "RemoveContainer" containerID="50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3"
Jul 08 19:36:25.163266 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:25.163247    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3\": container with ID starting with 50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3 not found: ID does not exist" containerID="50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3"
Jul 08 19:36:25.163315 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.163271    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3"} err="failed to get container status \"50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3\": rpc error: code = NotFound desc = could not find container \"50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3\": container with ID starting with 50b1541ee6ca112933876e7753c7f28cfe63d8108448741cfb6a9d792e382fb3 not found: ID does not exist"
Jul 08 19:36:25.168519 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.168499    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"]
Jul 08 19:36:25.173450 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:25.173418    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5fb4a8-openshift-default-7b5cc9fb5c-dzqg5"]
Jul 08 19:36:26.456457 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:26.456422    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" path="/var/lib/kubelet/pods/2296ae80-ddf8-420f-9f7c-d14e00fd9217/volumes"
Jul 08 19:36:26.628099 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:26.628057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:26.732664 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:26.732564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:26.919486 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:26.919454    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"]
Jul 08 19:36:26.919684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:26.919659    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy" containerID="cri-o://a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644" gracePeriod=30
Jul 08 19:36:26.932772 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:26.932739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:27.498041 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.498005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:27.795137 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.795061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"]
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796146    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796172    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796202    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796212    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796223    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f6a421-d50f-4c79-a266-6935e1e6f4b5" containerName="maas-api"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796232    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f6a421-d50f-4c79-a266-6935e1e6f4b5" containerName="maas-api"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796254    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4726f4a5-fdf2-423c-8c65-062fcd35793d" containerName="maas-api"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796262    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4726f4a5-fdf2-423c-8c65-062fcd35793d" containerName="maas-api"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796483    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f6a421-d50f-4c79-a266-6935e1e6f4b5" containerName="maas-api"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796501    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4726f4a5-fdf2-423c-8c65-062fcd35793d" containerName="maas-api"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796510    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy"
Jul 08 19:36:27.798323 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.796527    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2296ae80-ddf8-420f-9f7c-d14e00fd9217" containerName="istio-proxy"
Jul 08 19:36:27.805721 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.805696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.808760 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.808734    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3aa932-openshift-default-dockercfg-sth7c\""
Jul 08 19:36:27.811124 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.811097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"]
Jul 08 19:36:27.817659 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.817609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:27.946076 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-certs\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946243 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-token\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946243 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa531f-e728-4b48-829d-b107c3e2dd64-istiod-ca-cert\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946243 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-socket\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946243 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-podinfo\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946378 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-credential-socket\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946378 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48pf\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-kube-api-access-g48pf\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946448 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-envoy\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:27.946448 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:27.946403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-data\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.046941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-certs\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.046994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-token\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa531f-e728-4b48-829d-b107c3e2dd64-istiod-ca-cert\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-socket\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-podinfo\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-credential-socket\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48pf\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-kube-api-access-g48pf\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-envoy\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-data\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047403    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-certs\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047605 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047533    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-socket\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047782 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047747    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-data\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047916 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047835    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-credential-socket\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.047989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.047916    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa531f-e728-4b48-829d-b107c3e2dd64-istiod-ca-cert\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.049564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.049541    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-envoy\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.049788 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.049771    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-podinfo\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.055203 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.055181    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-token\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.055397 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.055378    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48pf\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-kube-api-access-g48pf\") pod \"e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") " pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.118791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.118764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:28.262770 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.262738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"]
Jul 08 19:36:28.263122 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:36:28.263085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aa531f_e728_4b48_829d_b107c3e2dd64.slice/crio-8b8e2dad2914575aeecb096c158519e400ebce56dad59b98cc3b39750288549a WatchSource:0}: Error finding container 8b8e2dad2914575aeecb096c158519e400ebce56dad59b98cc3b39750288549a: Status 404 returned error can't find the container with id 8b8e2dad2914575aeecb096c158519e400ebce56dad59b98cc3b39750288549a
Jul 08 19:36:28.265373 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.265346    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:28.265455 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.265441    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:28.265504 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.265474    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:28.321758 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:28.321729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:29.168442 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:29.168395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" event={"ID":"e3aa531f-e728-4b48-829d-b107c3e2dd64","Type":"ContainerStarted","Data":"296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd"}
Jul 08 19:36:29.168850 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:29.168455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" event={"ID":"e3aa531f-e728-4b48-829d-b107c3e2dd64","Type":"ContainerStarted","Data":"8b8e2dad2914575aeecb096c158519e400ebce56dad59b98cc3b39750288549a"}
Jul 08 19:36:29.195663 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:29.195598    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" podStartSLOduration=2.195582756 podStartE2EDuration="2.195582756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:36:29.192796812 +0000 UTC m=+2247.836817544" watchObservedRunningTime="2026-07-08 19:36:29.195582756 +0000 UTC m=+2247.839603485"
Jul 08 19:36:30.119312 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:30.119275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:30.124420 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:30.124397    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:30.174143 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:30.174110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:30.174980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:30.174954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:31.164848 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.164814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:31.235126 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.235086    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"]
Jul 08 19:36:31.240153 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.240130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.243529 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.243506    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3aa932-serving-cert\""
Jul 08 19:36:31.250559 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.250535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"]
Jul 08 19:36:31.267421 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.267391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:31.358244 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.358210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:31.380359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.380323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76e3983-f261-4ad1-8357-f4b68264bff1-maas-api-tls\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.380486 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.380450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfncm\" (UniqueName: \"kubernetes.io/projected/c76e3983-f261-4ad1-8357-f4b68264bff1-kube-api-access-mfncm\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.380562 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.380501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76e3983-f261-4ad1-8357-f4b68264bff1-openshift-service-ca\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.459145 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.459055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:31.481608 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.481576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76e3983-f261-4ad1-8357-f4b68264bff1-maas-api-tls\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.481794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.481697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfncm\" (UniqueName: \"kubernetes.io/projected/c76e3983-f261-4ad1-8357-f4b68264bff1-kube-api-access-mfncm\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.481794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.481754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76e3983-f261-4ad1-8357-f4b68264bff1-openshift-service-ca\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.482351 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.482327    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76e3983-f261-4ad1-8357-f4b68264bff1-openshift-service-ca\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.483894 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.483876    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76e3983-f261-4ad1-8357-f4b68264bff1-maas-api-tls\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.492966 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.492945    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfncm\" (UniqueName: \"kubernetes.io/projected/c76e3983-f261-4ad1-8357-f4b68264bff1-kube-api-access-mfncm\") pod \"maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.553084 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.553058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:31.697580 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.697545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"]
Jul 08 19:36:31.697580 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:36:31.697566    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76e3983_f261_4ad1_8357_f4b68264bff1.slice/crio-850bbf29de697a26276466c12f50a52d0719426b7ad5dadb25ec2455ca4f4f97 WatchSource:0}: Error finding container 850bbf29de697a26276466c12f50a52d0719426b7ad5dadb25ec2455ca4f4f97: Status 404 returned error can't find the container with id 850bbf29de697a26276466c12f50a52d0719426b7ad5dadb25ec2455ca4f4f97
Jul 08 19:36:31.761087 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.761055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:31.864794 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:31.864736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:32.158682 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.158657    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:36:32.187854 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.187820    2563 generic.go:356] "Generic (PLEG): container finished" podID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerID="a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644" exitCode=0
Jul 08 19:36:32.188021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.187872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"
Jul 08 19:36:32.188021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.187909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" event={"ID":"695d2159-f1e6-400d-9f5a-5180f68d7a14","Type":"ContainerDied","Data":"a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644"}
Jul 08 19:36:32.188021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.187972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg" event={"ID":"695d2159-f1e6-400d-9f5a-5180f68d7a14","Type":"ContainerDied","Data":"31ec99533321abaab306e68cc51421ca9d4d9ba1d77aacc93c209f89924d2626"}
Jul 08 19:36:32.188021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.188000    2563 scope.go:117] "RemoveContainer" containerID="a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644"
Jul 08 19:36:32.189699 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.189664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc" event={"ID":"c76e3983-f261-4ad1-8357-f4b68264bff1","Type":"ContainerStarted","Data":"850bbf29de697a26276466c12f50a52d0719426b7ad5dadb25ec2455ca4f4f97"}
Jul 08 19:36:32.198611 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.198592    2563 scope.go:117] "RemoveContainer" containerID="1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244"
Jul 08 19:36:32.208878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.208859    2563 scope.go:117] "RemoveContainer" containerID="a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644"
Jul 08 19:36:32.209155 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:32.209133    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644\": container with ID starting with a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644 not found: ID does not exist" containerID="a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644"
Jul 08 19:36:32.209219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.209164    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644"} err="failed to get container status \"a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644\": rpc error: code = NotFound desc = could not find container \"a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644\": container with ID starting with a5577bcb45762a7f1b83411312cee745b4abd883acf37e80d888a5bdffa4a644 not found: ID does not exist"
Jul 08 19:36:32.209219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.209183    2563 scope.go:117] "RemoveContainer" containerID="1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244"
Jul 08 19:36:32.209441 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:32.209420    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244\": container with ID starting with 1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244 not found: ID does not exist" containerID="1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244"
Jul 08 19:36:32.209486 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.209444    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244"} err="failed to get container status \"1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244\": rpc error: code = NotFound desc = could not find container \"1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244\": container with ID starting with 1e31dd34b5f593082fb3327d3e486381c4b3eff2d6f59d6ea262844313691244 not found: ID does not exist"
Jul 08 19:36:32.289352 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-envoy\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-data\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-podinfo\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkjn4\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-kube-api-access-wkjn4\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289471    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-credential-socket\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-token\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695d2159-f1e6-400d-9f5a-5180f68d7a14-istiod-ca-cert\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-certs\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.289779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-socket\") pod \"695d2159-f1e6-400d-9f5a-5180f68d7a14\" (UID: \"695d2159-f1e6-400d-9f5a-5180f68d7a14\") "
Jul 08 19:36:32.290189 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.289867    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:32.290189 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.290114    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.290337 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.290314    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695d2159-f1e6-400d-9f5a-5180f68d7a14-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:36:32.290424 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.290392    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-data" (OuterVolumeSpecName: "istio-data") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:32.290799 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.290776    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:32.290866 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.290802    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:32.291730 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.291703    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:36:32.291922 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.291892    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:36:32.292021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.291951    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-kube-api-access-wkjn4" (OuterVolumeSpecName: "kube-api-access-wkjn4") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "kube-api-access-wkjn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:36:32.292383 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.292363    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-token" (OuterVolumeSpecName: "istio-token") pod "695d2159-f1e6-400d-9f5a-5180f68d7a14" (UID: "695d2159-f1e6-400d-9f5a-5180f68d7a14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:36:32.390790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390759    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.390790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390783    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.390790 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390792    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.390984 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390801    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkjn4\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-kube-api-access-wkjn4\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.390984 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390809    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/695d2159-f1e6-400d-9f5a-5180f68d7a14-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.390984 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390818    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/695d2159-f1e6-400d-9f5a-5180f68d7a14-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.390984 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390825    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.390984 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.390833    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/695d2159-f1e6-400d-9f5a-5180f68d7a14-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:32.514230 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.514198    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"]
Jul 08 19:36:32.518527 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:32.518491    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fb0a08-openshift-default-f54dcf9b9-ncnkg"]
Jul 08 19:36:33.197322 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:33.197283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc" event={"ID":"c76e3983-f261-4ad1-8357-f4b68264bff1","Type":"ContainerStarted","Data":"81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0"}
Jul 08 19:36:33.197495 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:33.197436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:33.220669 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:33.220605    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc" podStartSLOduration=1.6784317130000002 podStartE2EDuration="2.220587775s" totalImagesPullingTime="542.156062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:36:33.218497599 +0000 UTC m=+2251.862518333" watchObservedRunningTime="2026-07-08 19:36:33.220587775 +0000 UTC m=+2251.864608506"
Jul 08 19:36:34.455397 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:34.455360    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" path="/var/lib/kubelet/pods/695d2159-f1e6-400d-9f5a-5180f68d7a14/volumes"
Jul 08 19:36:35.658722 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:35.658686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:36.122055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.122018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:36.425080 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.424994    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"]
Jul 08 19:36:36.425571 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.425554    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy"
Jul 08 19:36:36.425662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.425574    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy"
Jul 08 19:36:36.425662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.425597    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy"
Jul 08 19:36:36.425662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.425606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy"
Jul 08 19:36:36.425818 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.425711    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy"
Jul 08 19:36:36.425818 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.425726    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="695d2159-f1e6-400d-9f5a-5180f68d7a14" containerName="istio-proxy"
Jul 08 19:36:36.429250 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.429228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.432053 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.432036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-412f13-openshift-default-dockercfg-p9ftd\""
Jul 08 19:36:36.440833 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.440814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"]
Jul 08 19:36:36.459658 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.458335    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.528646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-credential-socket\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.528703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449c2246-2433-40e4-91aa-89ee95528c53-istio-podinfo\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.528734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-envoy\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.528817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-istio-token\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.528846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449c2246-2433-40e4-91aa-89ee95528c53-istiod-ca-cert\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.528938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-data\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.529019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-certs\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.529046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-socket\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.529466 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.529121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vdl\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-kube-api-access-r7vdl\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630373 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vdl\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-kube-api-access-r7vdl\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630373 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-credential-socket\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630606 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449c2246-2433-40e4-91aa-89ee95528c53-istio-podinfo\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630606 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-envoy\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630606 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-istio-token\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630606 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449c2246-2433-40e4-91aa-89ee95528c53-istiod-ca-cert\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630606 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-data\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630900 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-certs\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.630900 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-socket\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.631002 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.630935    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-credential-socket\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.631054 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.631012    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-certs\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.631054 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.631033    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-data\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.631258 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.631238    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449c2246-2433-40e4-91aa-89ee95528c53-istiod-ca-cert\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.631335 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.631275    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-socket\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.632832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.632810    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-envoy\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.633222 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.633199    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449c2246-2433-40e4-91aa-89ee95528c53-istio-podinfo\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.643464 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.643438    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vdl\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-kube-api-access-r7vdl\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.643899 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.643882    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-istio-token\") pod \"e2e-api-b-412f13-openshift-default-759fb54479-8cnqf\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") " pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.669293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.669258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:36.741320 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.741294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:36.874995 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.874968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"]
Jul 08 19:36:36.876251 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:36:36.876226    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449c2246_2433_40e4_91aa_89ee95528c53.slice/crio-166bdfbf766ad2cb84a985abfc9920057b6028ef87accd76bb50e60e196d2fc2 WatchSource:0}: Error finding container 166bdfbf766ad2cb84a985abfc9920057b6028ef87accd76bb50e60e196d2fc2: Status 404 returned error can't find the container with id 166bdfbf766ad2cb84a985abfc9920057b6028ef87accd76bb50e60e196d2fc2
Jul 08 19:36:36.878364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.878332    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:36.878446 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.878394    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:36.878446 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:36.878421    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:37.220402 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:37.220307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" event={"ID":"449c2246-2433-40e4-91aa-89ee95528c53","Type":"ContainerStarted","Data":"8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d"}
Jul 08 19:36:37.220402 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:37.220372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" event={"ID":"449c2246-2433-40e4-91aa-89ee95528c53","Type":"ContainerStarted","Data":"166bdfbf766ad2cb84a985abfc9920057b6028ef87accd76bb50e60e196d2fc2"}
Jul 08 19:36:37.305801 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:37.305751    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" podStartSLOduration=1.3057341230000001 podStartE2EDuration="1.305734123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:36:37.303793484 +0000 UTC m=+2255.947814214" watchObservedRunningTime="2026-07-08 19:36:37.305734123 +0000 UTC m=+2255.949754852"
Jul 08 19:36:37.741517 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:37.741482    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:37.746032 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:37.746004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:38.226235 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:38.226160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:38.227016 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:38.226998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:39.208190 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.208164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:39.271283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.271241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:39.371335 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.371302    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"]
Jul 08 19:36:39.375283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.375266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.378783 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.378760    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-412f13-serving-cert\""
Jul 08 19:36:39.385193 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.385170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"]
Jul 08 19:36:39.399978 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.399936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:39.458429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.458351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce3cbb2-a2b1-488c-986a-9e247aa44224-openshift-service-ca\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.458429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.458406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.458576 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.458558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpw8\" (UniqueName: \"kubernetes.io/projected/dce3cbb2-a2b1-488c-986a-9e247aa44224-kube-api-access-fvpw8\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.487258 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.487226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:39.559901 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.559872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.560097 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:39.559975    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-serving-cert: secret "maas-api-e2e-api-b-412f13-serving-cert" not found
Jul 08 19:36:39.560097 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:39.560060    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls podName:dce3cbb2-a2b1-488c-986a-9e247aa44224 nodeName:}" failed. No retries permitted until 2026-07-08 19:36:40.060038094 +0000 UTC m=+2258.704058803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls") pod "maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m" (UID: "dce3cbb2-a2b1-488c-986a-9e247aa44224") : secret "maas-api-e2e-api-b-412f13-serving-cert" not found
Jul 08 19:36:39.560364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.560340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpw8\" (UniqueName: \"kubernetes.io/projected/dce3cbb2-a2b1-488c-986a-9e247aa44224-kube-api-access-fvpw8\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.560452 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.560398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce3cbb2-a2b1-488c-986a-9e247aa44224-openshift-service-ca\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.561012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.560995    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce3cbb2-a2b1-488c-986a-9e247aa44224-openshift-service-ca\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.570087 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.570059    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpw8\" (UniqueName: \"kubernetes.io/projected/dce3cbb2-a2b1-488c-986a-9e247aa44224-kube-api-access-fvpw8\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:39.971111 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:39.971078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:40.066127 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:40.066094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:40.068520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:40.068495    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls\") pod \"maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:40.071779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:40.071751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:40.288776 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:40.288724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:40.427956 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:40.427921    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"]
Jul 08 19:36:40.428439 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:36:40.428412    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce3cbb2_a2b1_488c_986a_9e247aa44224.slice/crio-48ff44d523940f68e727f29464bdc08d8bffba174e498fd1e3f12e0b4358273b WatchSource:0}: Error finding container 48ff44d523940f68e727f29464bdc08d8bffba174e498fd1e3f12e0b4358273b: Status 404 returned error can't find the container with id 48ff44d523940f68e727f29464bdc08d8bffba174e498fd1e3f12e0b4358273b
Jul 08 19:36:41.247999 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:41.247898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m" event={"ID":"dce3cbb2-a2b1-488c-986a-9e247aa44224","Type":"ContainerStarted","Data":"22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5"}
Jul 08 19:36:41.248163 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:41.248005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m" event={"ID":"dce3cbb2-a2b1-488c-986a-9e247aa44224","Type":"ContainerStarted","Data":"48ff44d523940f68e727f29464bdc08d8bffba174e498fd1e3f12e0b4358273b"}
Jul 08 19:36:41.250225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:41.248687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:41.270028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:41.269971    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m" podStartSLOduration=1.702941331 podStartE2EDuration="2.269954582s" totalImagesPullingTime="567.013251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:36:41.268044548 +0000 UTC m=+2259.912065281" watchObservedRunningTime="2026-07-08 19:36:41.269954582 +0000 UTC m=+2259.913975332"
Jul 08 19:36:43.790125 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:43.790088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:44.865660 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:44.865599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:46.215779 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:46.215739    2563 patch_prober.go:28] interesting pod/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 19:36:46.216230 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:46.215800    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 19:36:47.263324 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:47.263302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:36:47.290072 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:47.290039    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerID="296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd" exitCode=0
Jul 08 19:36:47.290176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:47.290114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" event={"ID":"e3aa531f-e728-4b48-829d-b107c3e2dd64","Type":"ContainerDied","Data":"296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd"}
Jul 08 19:36:47.290541 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:47.290525    2563 scope.go:117] "RemoveContainer" containerID="296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd"
Jul 08 19:36:47.291723 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:47.291690    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:47.291843 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:47.291769    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:47.291843 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:47.291809    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:48.297586 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:48.297551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" event={"ID":"e3aa531f-e728-4b48-829d-b107c3e2dd64","Type":"ContainerStarted","Data":"3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53"}
Jul 08 19:36:49.119354 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:49.119319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:49.119537 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:49.119490    2563 patch_prober.go:28] interesting pod/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 08 19:36:49.119583 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:49.119531    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 08 19:36:50.120000 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:50.119967    2563 patch_prober.go:28] interesting pod/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 08 19:36:50.120530 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:50.120501    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 08 19:36:51.120057 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:51.120019    2563 patch_prober.go:28] interesting pod/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 08 19:36:51.120454 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:51.120072    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 08 19:36:51.961646 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:51.957925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:52.119393 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.119356    2563 patch_prober.go:28] interesting pod/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 08 19:36:52.119590 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.119429    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 08 19:36:52.306807 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.306726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:52.483091 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.483054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:52.582351 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.582256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:52.770560 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.770512    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"]
Jul 08 19:36:52.771418 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.771375    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc" podUID="c76e3983-f261-4ad1-8357-f4b68264bff1" containerName="maas-api" containerID="cri-o://81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0" gracePeriod=30
Jul 08 19:36:52.783894 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:52.783869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:53.031073 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.031052    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:53.105108 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.105078    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76e3983-f261-4ad1-8357-f4b68264bff1-maas-api-tls\") pod \"c76e3983-f261-4ad1-8357-f4b68264bff1\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") "
Jul 08 19:36:53.105275 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.105257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfncm\" (UniqueName: \"kubernetes.io/projected/c76e3983-f261-4ad1-8357-f4b68264bff1-kube-api-access-mfncm\") pod \"c76e3983-f261-4ad1-8357-f4b68264bff1\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") "
Jul 08 19:36:53.105396 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.105381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76e3983-f261-4ad1-8357-f4b68264bff1-openshift-service-ca\") pod \"c76e3983-f261-4ad1-8357-f4b68264bff1\" (UID: \"c76e3983-f261-4ad1-8357-f4b68264bff1\") "
Jul 08 19:36:53.105751 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.105727    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76e3983-f261-4ad1-8357-f4b68264bff1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c76e3983-f261-4ad1-8357-f4b68264bff1" (UID: "c76e3983-f261-4ad1-8357-f4b68264bff1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:36:53.107140 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.107119    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76e3983-f261-4ad1-8357-f4b68264bff1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c76e3983-f261-4ad1-8357-f4b68264bff1" (UID: "c76e3983-f261-4ad1-8357-f4b68264bff1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:36:53.107230 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.107178    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76e3983-f261-4ad1-8357-f4b68264bff1-kube-api-access-mfncm" (OuterVolumeSpecName: "kube-api-access-mfncm") pod "c76e3983-f261-4ad1-8357-f4b68264bff1" (UID: "c76e3983-f261-4ad1-8357-f4b68264bff1"). InnerVolumeSpecName "kube-api-access-mfncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:36:53.122333 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.122311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:53.122673 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.122654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:53.123238 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.123218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:36:53.207024 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.206924    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76e3983-f261-4ad1-8357-f4b68264bff1-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:53.207024 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.206957    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c76e3983-f261-4ad1-8357-f4b68264bff1-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:53.207024 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.206974    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfncm\" (UniqueName: \"kubernetes.io/projected/c76e3983-f261-4ad1-8357-f4b68264bff1-kube-api-access-mfncm\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:36:53.326022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.325993    2563 generic.go:356] "Generic (PLEG): container finished" podID="c76e3983-f261-4ad1-8357-f4b68264bff1" containerID="81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0" exitCode=0
Jul 08 19:36:53.326412 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.326041    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"
Jul 08 19:36:53.326412 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.326082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc" event={"ID":"c76e3983-f261-4ad1-8357-f4b68264bff1","Type":"ContainerDied","Data":"81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0"}
Jul 08 19:36:53.326412 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.326136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc" event={"ID":"c76e3983-f261-4ad1-8357-f4b68264bff1","Type":"ContainerDied","Data":"850bbf29de697a26276466c12f50a52d0719426b7ad5dadb25ec2455ca4f4f97"}
Jul 08 19:36:53.326412 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.326168    2563 scope.go:117] "RemoveContainer" containerID="81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0"
Jul 08 19:36:53.336172 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.336156    2563 scope.go:117] "RemoveContainer" containerID="81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0"
Jul 08 19:36:53.336411 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:36:53.336392    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0\": container with ID starting with 81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0 not found: ID does not exist" containerID="81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0"
Jul 08 19:36:53.336467 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.336419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0"} err="failed to get container status \"81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0\": rpc error: code = NotFound desc = could not find container \"81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0\": container with ID starting with 81b6e26d7364743ae06052928b967c21c957290f1315926acb01fb1b453e78f0 not found: ID does not exist"
Jul 08 19:36:53.349765 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.349743    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"]
Jul 08 19:36:53.354477 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:53.354457    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3aa932-d75d598c4-x2pzc"]
Jul 08 19:36:54.267808 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:54.267767    2563 patch_prober.go:28] interesting pod/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 08 19:36:54.267989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:54.267835    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 19:36:54.456074 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:54.456037    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76e3983-f261-4ad1-8357-f4b68264bff1" path="/var/lib/kubelet/pods/c76e3983-f261-4ad1-8357-f4b68264bff1/volumes"
Jul 08 19:36:56.346066 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:56.346036    2563 generic.go:356] "Generic (PLEG): container finished" podID="449c2246-2433-40e4-91aa-89ee95528c53" containerID="8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d" exitCode=0
Jul 08 19:36:56.346468 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:56.346105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" event={"ID":"449c2246-2433-40e4-91aa-89ee95528c53","Type":"ContainerDied","Data":"8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d"}
Jul 08 19:36:56.346529 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:56.346514    2563 scope.go:117] "RemoveContainer" containerID="8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d"
Jul 08 19:36:56.347679 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:56.347612    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:56.347784 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:56.347730    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:56.347837 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:56.347784    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:36:57.358522 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:57.358481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" event={"ID":"449c2246-2433-40e4-91aa-89ee95528c53","Type":"ContainerStarted","Data":"2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8"}
Jul 08 19:36:57.741755 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:57.741719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:36:57.741950 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:57.741847    2563 patch_prober.go:28] interesting pod/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 08 19:36:57.741950 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:57.741890    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 08 19:36:58.155496 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.155405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:58.221880 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.221848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:58.414887 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.414809    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"]
Jul 08 19:36:58.415304 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.415073    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy" containerID="cri-o://3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53" gracePeriod=30
Jul 08 19:36:58.432100 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.432056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:58.742478 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.742444    2563 patch_prober.go:28] interesting pod/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 08 19:36:58.742763 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.742732    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 08 19:36:58.903913 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:58.903872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:59.372175 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.372140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:59.561647 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.561467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:59.662518 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.662403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:36:59.742330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.742291    2563 patch_prober.go:28] interesting pod/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 08 19:36:59.742535 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.742362    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 08 19:36:59.847650 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.842337    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"]
Jul 08 19:36:59.847650 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.842563    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m" podUID="dce3cbb2-a2b1-488c-986a-9e247aa44224" containerName="maas-api" containerID="cri-o://22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5" gracePeriod=30
Jul 08 19:36:59.864167 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:36:59.864130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:00.120310 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.120285    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:37:00.286924 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.286892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls\") pod \"dce3cbb2-a2b1-488c-986a-9e247aa44224\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") "
Jul 08 19:37:00.287075 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.286990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvpw8\" (UniqueName: \"kubernetes.io/projected/dce3cbb2-a2b1-488c-986a-9e247aa44224-kube-api-access-fvpw8\") pod \"dce3cbb2-a2b1-488c-986a-9e247aa44224\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") "
Jul 08 19:37:00.287075 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.287017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce3cbb2-a2b1-488c-986a-9e247aa44224-openshift-service-ca\") pod \"dce3cbb2-a2b1-488c-986a-9e247aa44224\" (UID: \"dce3cbb2-a2b1-488c-986a-9e247aa44224\") "
Jul 08 19:37:00.287385 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.287361    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce3cbb2-a2b1-488c-986a-9e247aa44224-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dce3cbb2-a2b1-488c-986a-9e247aa44224" (UID: "dce3cbb2-a2b1-488c-986a-9e247aa44224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:37:00.289022 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.288997    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dce3cbb2-a2b1-488c-986a-9e247aa44224" (UID: "dce3cbb2-a2b1-488c-986a-9e247aa44224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:37:00.289108 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.289095    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce3cbb2-a2b1-488c-986a-9e247aa44224-kube-api-access-fvpw8" (OuterVolumeSpecName: "kube-api-access-fvpw8") pod "dce3cbb2-a2b1-488c-986a-9e247aa44224" (UID: "dce3cbb2-a2b1-488c-986a-9e247aa44224"). InnerVolumeSpecName "kube-api-access-fvpw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:37:00.377092 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.377058    2563 generic.go:356] "Generic (PLEG): container finished" podID="dce3cbb2-a2b1-488c-986a-9e247aa44224" containerID="22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5" exitCode=0
Jul 08 19:37:00.377249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.377101    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"
Jul 08 19:37:00.377249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.377148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m" event={"ID":"dce3cbb2-a2b1-488c-986a-9e247aa44224","Type":"ContainerDied","Data":"22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5"}
Jul 08 19:37:00.377249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.377199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m" event={"ID":"dce3cbb2-a2b1-488c-986a-9e247aa44224","Type":"ContainerDied","Data":"48ff44d523940f68e727f29464bdc08d8bffba174e498fd1e3f12e0b4358273b"}
Jul 08 19:37:00.377249 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.377222    2563 scope.go:117] "RemoveContainer" containerID="22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5"
Jul 08 19:37:00.386572 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.386552    2563 scope.go:117] "RemoveContainer" containerID="22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5"
Jul 08 19:37:00.386850 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:00.386833    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5\": container with ID starting with 22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5 not found: ID does not exist" containerID="22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5"
Jul 08 19:37:00.386908 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.386859    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5"} err="failed to get container status \"22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5\": rpc error: code = NotFound desc = could not find container \"22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5\": container with ID starting with 22ec88a61735f115d7b8c9bf7e04c45802028ebee0c96bf35b8633572352d5f5 not found: ID does not exist"
Jul 08 19:37:00.388288 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.388272    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce3cbb2-a2b1-488c-986a-9e247aa44224-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:00.388330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.388292    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce3cbb2-a2b1-488c-986a-9e247aa44224-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:00.388330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.388300    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvpw8\" (UniqueName: \"kubernetes.io/projected/dce3cbb2-a2b1-488c-986a-9e247aa44224-kube-api-access-fvpw8\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:00.400120 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.400087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"]
Jul 08 19:37:00.403792 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.403772    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-412f13-64f97fdb5f-nvr2m"]
Jul 08 19:37:00.459662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.459607    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce3cbb2-a2b1-488c-986a-9e247aa44224" path="/var/lib/kubelet/pods/dce3cbb2-a2b1-488c-986a-9e247aa44224/volumes"
Jul 08 19:37:00.742895 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.742864    2563 patch_prober.go:28] interesting pod/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 08 19:37:00.743465 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:00.743432    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 08 19:37:01.745914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:01.745885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:37:01.746302 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:01.746263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:37:01.746953 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:01.746933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:37:02.834936 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:02.834850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:02.934610 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:02.934575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:03.673488 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.673467    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:37:03.822297 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-token\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822445 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa531f-e728-4b48-829d-b107c3e2dd64-istiod-ca-cert\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822445 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g48pf\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-kube-api-access-g48pf\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822445 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-credential-socket\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822445 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-podinfo\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822655 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-envoy\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822655 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-data\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822655 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-socket\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822655 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-certs\") pod \"e3aa531f-e728-4b48-829d-b107c3e2dd64\" (UID: \"e3aa531f-e728-4b48-829d-b107c3e2dd64\") "
Jul 08 19:37:03.822871 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822700    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:03.822947 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.822832    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aa531f-e728-4b48-829d-b107c3e2dd64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:37:03.823258 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.823075    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3aa531f-e728-4b48-829d-b107c3e2dd64-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.823258 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.823097    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.823258 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.823122    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:03.823258 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.823220    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:03.823608 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.823470    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:03.824753 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.824721    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:03.824878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.824856    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-kube-api-access-g48pf" (OuterVolumeSpecName: "kube-api-access-g48pf") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "kube-api-access-g48pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:37:03.824973 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.824956    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:37:03.825009 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.824974    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3aa531f-e728-4b48-829d-b107c3e2dd64" (UID: "e3aa531f-e728-4b48-829d-b107c3e2dd64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:37:03.924293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.924258    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48pf\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-kube-api-access-g48pf\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.924293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.924282    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.924293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.924292    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.924293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.924300    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.924789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.924309    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.924789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.924316    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3aa531f-e728-4b48-829d-b107c3e2dd64-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:03.924789 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:03.924324    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3aa531f-e728-4b48-829d-b107c3e2dd64-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:04.400070 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.400040    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerID="3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53" exitCode=0
Jul 08 19:37:04.400239 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.400092    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"
Jul 08 19:37:04.400239 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.400119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" event={"ID":"e3aa531f-e728-4b48-829d-b107c3e2dd64","Type":"ContainerDied","Data":"3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53"}
Jul 08 19:37:04.400239 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.400174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c" event={"ID":"e3aa531f-e728-4b48-829d-b107c3e2dd64","Type":"ContainerDied","Data":"8b8e2dad2914575aeecb096c158519e400ebce56dad59b98cc3b39750288549a"}
Jul 08 19:37:04.400239 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.400199    2563 scope.go:117] "RemoveContainer" containerID="3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53"
Jul 08 19:37:04.409890 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.409872    2563 scope.go:117] "RemoveContainer" containerID="296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd"
Jul 08 19:37:04.418295 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.418275    2563 scope.go:117] "RemoveContainer" containerID="3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53"
Jul 08 19:37:04.418524 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:04.418506    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53\": container with ID starting with 3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53 not found: ID does not exist" containerID="3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53"
Jul 08 19:37:04.418588 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.418534    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53"} err="failed to get container status \"3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53\": rpc error: code = NotFound desc = could not find container \"3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53\": container with ID starting with 3abfbaf05db94a38916ef7241578a78a09970217e4cca06819e2353cc3961a53 not found: ID does not exist"
Jul 08 19:37:04.418588 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.418550    2563 scope.go:117] "RemoveContainer" containerID="296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd"
Jul 08 19:37:04.418799 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:04.418764    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd\": container with ID starting with 296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd not found: ID does not exist" containerID="296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd"
Jul 08 19:37:04.418799 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.418785    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd"} err="failed to get container status \"296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd\": rpc error: code = NotFound desc = could not find container \"296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd\": container with ID starting with 296912a56cac538ac6321a1b74a4f843d3178fb942aad4c7a8a31cd1c77744fd not found: ID does not exist"
Jul 08 19:37:04.424074 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.424052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"]
Jul 08 19:37:04.427773 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.427752    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3aa932-openshift-default-565bffcc9c-5h97c"]
Jul 08 19:37:04.455672 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:04.455646    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" path="/var/lib/kubelet/pods/e3aa531f-e728-4b48-829d-b107c3e2dd64/volumes"
Jul 08 19:37:05.283165 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:05.283127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:05.349845 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:05.349802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:05.532117 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:05.532084    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"]
Jul 08 19:37:05.532342 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:05.532317    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy" containerID="cri-o://2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8" gracePeriod=30
Jul 08 19:37:05.554066 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:05.553986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:06.195521 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.195491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:06.425020 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.424989    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"]
Jul 08 19:37:06.425522 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76e3983-f261-4ad1-8357-f4b68264bff1" containerName="maas-api"
Jul 08 19:37:06.425604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425525    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76e3983-f261-4ad1-8357-f4b68264bff1" containerName="maas-api"
Jul 08 19:37:06.425604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425536    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce3cbb2-a2b1-488c-986a-9e247aa44224" containerName="maas-api"
Jul 08 19:37:06.425604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425544    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce3cbb2-a2b1-488c-986a-9e247aa44224" containerName="maas-api"
Jul 08 19:37:06.425604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425566    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy"
Jul 08 19:37:06.425604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425575    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy"
Jul 08 19:37:06.425604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425598    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy"
Jul 08 19:37:06.425604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy"
Jul 08 19:37:06.425951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425724    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy"
Jul 08 19:37:06.425951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425739    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce3cbb2-a2b1-488c-986a-9e247aa44224" containerName="maas-api"
Jul 08 19:37:06.425951 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.425750    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76e3983-f261-4ad1-8357-f4b68264bff1" containerName="maas-api"
Jul 08 19:37:06.428912 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.428892    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.431823 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.431803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-866a5b-openshift-default-dockercfg-ftfs9\""
Jul 08 19:37:06.441648 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.441600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"]
Jul 08 19:37:06.447231 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.447171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:06.550069 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-podinfo\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550248 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-certs\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550248 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-socket\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550248 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-envoy\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550358 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-data\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550358 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-token\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550358 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcxb\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-kube-api-access-lqcxb\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550449 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7143a01-12ff-4547-ac65-e355f2b555ef-istiod-ca-cert\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.550449 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.550402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-credential-socket\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-data\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-token\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651437 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcxb\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-kube-api-access-lqcxb\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651437 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7143a01-12ff-4547-ac65-e355f2b555ef-istiod-ca-cert\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651437 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-credential-socket\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651437 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-podinfo\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-certs\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-socket\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-envoy\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651684 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651672    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-data\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651885 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651786    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-credential-socket\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651931 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651888    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-certs\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.651989 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.651960    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-socket\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.652105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.652078    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7143a01-12ff-4547-ac65-e355f2b555ef-istiod-ca-cert\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.653770 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.653748    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-envoy\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.653893 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.653877    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-podinfo\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.659976 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.659954    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-token\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.660339 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.660315    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcxb\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-kube-api-access-lqcxb\") pod \"e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") " pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.740852 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.740832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:06.879033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.879005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"]
Jul 08 19:37:06.880040 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:37:06.880017    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7143a01_12ff_4547_ac65_e355f2b555ef.slice/crio-73cc1f4331c17cfd080e420ae3d16b650d25ca479104a8b50f6ac9f2ab3580cd WatchSource:0}: Error finding container 73cc1f4331c17cfd080e420ae3d16b650d25ca479104a8b50f6ac9f2ab3580cd: Status 404 returned error can't find the container with id 73cc1f4331c17cfd080e420ae3d16b650d25ca479104a8b50f6ac9f2ab3580cd
Jul 08 19:37:06.882800 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.882735    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:06.882915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.882823    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:06.882915 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:06.882850    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:07.054662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:07.054548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:07.421971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:07.421882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" event={"ID":"e7143a01-12ff-4547-ac65-e355f2b555ef","Type":"ContainerStarted","Data":"98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be"}
Jul 08 19:37:07.421971 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:07.421935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" event={"ID":"e7143a01-12ff-4547-ac65-e355f2b555ef","Type":"ContainerStarted","Data":"73cc1f4331c17cfd080e420ae3d16b650d25ca479104a8b50f6ac9f2ab3580cd"}
Jul 08 19:37:07.448811 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:07.448760    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" podStartSLOduration=1.448744413 podStartE2EDuration="1.448744413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:37:07.445964019 +0000 UTC m=+2286.089984750" watchObservedRunningTime="2026-07-08 19:37:07.448744413 +0000 UTC m=+2286.092765177"
Jul 08 19:37:07.741693 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:07.741670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:07.743165 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:07.743133    2563 patch_prober.go:28] interesting pod/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 08 19:37:07.743321 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:07.743182    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 08 19:37:08.744921 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:08.744893    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:09.401922 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.401874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:09.432572 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.432538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:09.433395 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.433373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:09.471309 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.471279    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"]
Jul 08 19:37:09.471832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.471806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3aa531f-e728-4b48-829d-b107c3e2dd64" containerName="istio-proxy"
Jul 08 19:37:09.474237 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.474218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.479104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.478229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrf9\" (UniqueName: \"kubernetes.io/projected/c477f4be-db06-4d16-91b1-98e1671e3b65-kube-api-access-xvrf9\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.479104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.478316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c477f4be-db06-4d16-91b1-98e1671e3b65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.479104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.478400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.479104 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.478617    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-866a5b-serving-cert\""
Jul 08 19:37:09.491485 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.490745    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"]
Jul 08 19:37:09.491485 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.491338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:09.580190 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.580162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrf9\" (UniqueName: \"kubernetes.io/projected/c477f4be-db06-4d16-91b1-98e1671e3b65-kube-api-access-xvrf9\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.580378 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.580215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c477f4be-db06-4d16-91b1-98e1671e3b65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.580378 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.580255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.580505 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:09.580426    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-serving-cert: secret "maas-api-e2e-shared-a-866a5b-serving-cert" not found
Jul 08 19:37:09.580564 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:09.580519    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls podName:c477f4be-db06-4d16-91b1-98e1671e3b65 nodeName:}" failed. No retries permitted until 2026-07-08 19:37:10.080481298 +0000 UTC m=+2288.724502012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls") pod "maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c" (UID: "c477f4be-db06-4d16-91b1-98e1671e3b65") : secret "maas-api-e2e-shared-a-866a5b-serving-cert" not found
Jul 08 19:37:09.581211 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.581179    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c477f4be-db06-4d16-91b1-98e1671e3b65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.590126 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.590095    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrf9\" (UniqueName: \"kubernetes.io/projected/c477f4be-db06-4d16-91b1-98e1671e3b65-kube-api-access-xvrf9\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:09.590261 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.590165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:09.963911 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:09.963876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:10.066179 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.066144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:10.085471 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.085435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:10.088027 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.088007    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls\") pod \"maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:10.091840 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.091819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:10.165462 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.165395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:10.246466 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.246435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"]
Jul 08 19:37:10.247098 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:37:10.247073    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc477f4be_db06_4d16_91b1_98e1671e3b65.slice/crio-a886c52e48600c9986aa82b10d2ea68103809263d5e4b47ec2f97ec0890ed9b6 WatchSource:0}: Error finding container a886c52e48600c9986aa82b10d2ea68103809263d5e4b47ec2f97ec0890ed9b6: Status 404 returned error can't find the container with id a886c52e48600c9986aa82b10d2ea68103809263d5e4b47ec2f97ec0890ed9b6
Jul 08 19:37:10.439121 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.439089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c" event={"ID":"c477f4be-db06-4d16-91b1-98e1671e3b65","Type":"ContainerStarted","Data":"a886c52e48600c9986aa82b10d2ea68103809263d5e4b47ec2f97ec0890ed9b6"}
Jul 08 19:37:10.789255 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.789231    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:37:10.893783 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.893692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-socket\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.893942 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.893798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449c2246-2433-40e4-91aa-89ee95528c53-istiod-ca-cert\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.893942 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.893862    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7vdl\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-kube-api-access-r7vdl\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.894071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.893975    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:10.894071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.893993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-credential-socket\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.894071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894036    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-data\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.894071 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-certs\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.894259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-istio-token\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.894259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-envoy\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.894259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894168    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449c2246-2433-40e4-91aa-89ee95528c53-istio-podinfo\") pod \"449c2246-2433-40e4-91aa-89ee95528c53\" (UID: \"449c2246-2433-40e4-91aa-89ee95528c53\") "
Jul 08 19:37:10.894259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894171    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:10.894259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894181    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449c2246-2433-40e4-91aa-89ee95528c53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:37:10.894498 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894357    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:10.894750 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894728    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/449c2246-2433-40e4-91aa-89ee95528c53-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.894856 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894753    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.894856 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894767    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.894856 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.894778    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.895080 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.895060    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-data" (OuterVolumeSpecName: "istio-data") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:10.896798 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.896777    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:37:10.896881 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.896795    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-istio-token" (OuterVolumeSpecName: "istio-token") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:37:10.896921 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.896887    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/449c2246-2433-40e4-91aa-89ee95528c53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:37:10.896921 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.896903    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-kube-api-access-r7vdl" (OuterVolumeSpecName: "kube-api-access-r7vdl") pod "449c2246-2433-40e4-91aa-89ee95528c53" (UID: "449c2246-2433-40e4-91aa-89ee95528c53"). InnerVolumeSpecName "kube-api-access-r7vdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:37:10.995853 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.995817    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vdl\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-kube-api-access-r7vdl\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.995853 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.995853    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.996254 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.995869    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449c2246-2433-40e4-91aa-89ee95528c53-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.996254 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.995881    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/449c2246-2433-40e4-91aa-89ee95528c53-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:10.996254 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:10.995895    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/449c2246-2433-40e4-91aa-89ee95528c53-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:37:11.445700 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.445665    2563 generic.go:356] "Generic (PLEG): container finished" podID="449c2246-2433-40e4-91aa-89ee95528c53" containerID="2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8" exitCode=0
Jul 08 19:37:11.445879 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.445719    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"
Jul 08 19:37:11.445879 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.445752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" event={"ID":"449c2246-2433-40e4-91aa-89ee95528c53","Type":"ContainerDied","Data":"2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8"}
Jul 08 19:37:11.445879 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.445810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf" event={"ID":"449c2246-2433-40e4-91aa-89ee95528c53","Type":"ContainerDied","Data":"166bdfbf766ad2cb84a985abfc9920057b6028ef87accd76bb50e60e196d2fc2"}
Jul 08 19:37:11.445879 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.445837    2563 scope.go:117] "RemoveContainer" containerID="2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8"
Jul 08 19:37:11.447334 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.447311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c" event={"ID":"c477f4be-db06-4d16-91b1-98e1671e3b65","Type":"ContainerStarted","Data":"1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602"}
Jul 08 19:37:11.447417 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.447365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:11.455039 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.455025    2563 scope.go:117] "RemoveContainer" containerID="8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d"
Jul 08 19:37:11.462761 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.462744    2563 scope.go:117] "RemoveContainer" containerID="2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8"
Jul 08 19:37:11.463035 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:11.463016    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8\": container with ID starting with 2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8 not found: ID does not exist" containerID="2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8"
Jul 08 19:37:11.463107 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.463043    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8"} err="failed to get container status \"2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8\": rpc error: code = NotFound desc = could not find container \"2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8\": container with ID starting with 2029a362e9ae290ba113861b34f25fbfce701c2435a1a89c2604b5f8b46579f8 not found: ID does not exist"
Jul 08 19:37:11.463107 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.463060    2563 scope.go:117] "RemoveContainer" containerID="8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d"
Jul 08 19:37:11.463298 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:11.463273    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d\": container with ID starting with 8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d not found: ID does not exist" containerID="8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d"
Jul 08 19:37:11.463338 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.463302    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d"} err="failed to get container status \"8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d\": rpc error: code = NotFound desc = could not find container \"8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d\": container with ID starting with 8e67687fffffe25b9832662a6366c49ffff552b9af9e674b58fc1855bde8cd1d not found: ID does not exist"
Jul 08 19:37:11.471947 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.471903    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c" podStartSLOduration=2.058086521 podStartE2EDuration="2.471889899s" totalImagesPullingTime="413.803378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:37:11.470511805 +0000 UTC m=+2290.114532536" watchObservedRunningTime="2026-07-08 19:37:11.471889899 +0000 UTC m=+2290.115910632"
Jul 08 19:37:11.486176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.486151    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"]
Jul 08 19:37:11.488472 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:11.488453    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-412f13-openshift-default-759fb54479-8cnqf"]
Jul 08 19:37:12.456311 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:12.456276    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449c2246-2433-40e4-91aa-89ee95528c53" path="/var/lib/kubelet/pods/449c2246-2433-40e4-91aa-89ee95528c53/volumes"
Jul 08 19:37:13.890893 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:13.890860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:14.210941 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.210846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:14.497038 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.497006    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"]
Jul 08 19:37:14.497520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.497500    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy"
Jul 08 19:37:14.497520 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.497521    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy"
Jul 08 19:37:14.497711 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.497557    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy"
Jul 08 19:37:14.497711 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.497565    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy"
Jul 08 19:37:14.497711 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.497709    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy"
Jul 08 19:37:14.500436 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.500414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.503330 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.503309    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e3bca4-openshift-default-dockercfg-ksm6h\""
Jul 08 19:37:14.511021 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.510988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"]
Jul 08 19:37:14.526199 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.526170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:14.530248 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-credential-socket\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-envoy\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-socket\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530374 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-certs\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530558 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-data\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530558 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09613d34-6c30-4753-b47f-da8de162a37d-istio-podinfo\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530558 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjq4c\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-kube-api-access-wjq4c\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09613d34-6c30-4753-b47f-da8de162a37d-istiod-ca-cert\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.530741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.530684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-istio-token\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.631949 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.631915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-credential-socket\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.631949 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.631953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-envoy\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.631984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-socket\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-certs\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-data\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09613d34-6c30-4753-b47f-da8de162a37d-istio-podinfo\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632424 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjq4c\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-kube-api-access-wjq4c\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632424 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09613d34-6c30-4753-b47f-da8de162a37d-istiod-ca-cert\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632424 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632359    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-socket\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632424 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-istio-token\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632424 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632415    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-certs\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632460    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-credential-socket\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.632904 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.632885    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-data\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.633187 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.633145    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09613d34-6c30-4753-b47f-da8de162a37d-istiod-ca-cert\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.634331 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.634310    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-envoy\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.634705 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.634686    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09613d34-6c30-4753-b47f-da8de162a37d-istio-podinfo\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.641432 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.641400    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-istio-token\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.641432 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.641417    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjq4c\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-kube-api-access-wjq4c\") pod \"e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") " pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.812888 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.812791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:14.955033 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.954992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"]
Jul 08 19:37:14.956601 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:37:14.956574    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09613d34_6c30_4753_b47f_da8de162a37d.slice/crio-37b5afb1245cb333fdcc210423fcbf0243a89950c6404571cd5039976ba127f3 WatchSource:0}: Error finding container 37b5afb1245cb333fdcc210423fcbf0243a89950c6404571cd5039976ba127f3: Status 404 returned error can't find the container with id 37b5afb1245cb333fdcc210423fcbf0243a89950c6404571cd5039976ba127f3
Jul 08 19:37:14.958874 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.958842    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:14.958947 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.958905    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:14.958947 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:14.958933    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:15.036535 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:15.036511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:15.471839 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:15.471808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" event={"ID":"09613d34-6c30-4753-b47f-da8de162a37d","Type":"ContainerStarted","Data":"4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02"}
Jul 08 19:37:15.472018 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:15.471859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" event={"ID":"09613d34-6c30-4753-b47f-da8de162a37d","Type":"ContainerStarted","Data":"37b5afb1245cb333fdcc210423fcbf0243a89950c6404571cd5039976ba127f3"}
Jul 08 19:37:15.496645 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:15.496564    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" podStartSLOduration=1.496548703 podStartE2EDuration="1.496548703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:37:15.49606165 +0000 UTC m=+2294.140082391" watchObservedRunningTime="2026-07-08 19:37:15.496548703 +0000 UTC m=+2294.140569432"
Jul 08 19:37:15.813213 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:15.813176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:16.818456 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:16.818426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:17.436480 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.436441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:17.468253 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.468220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:37:17.484034 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.484005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:17.484935 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.484920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:17.513961 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.513931    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"]
Jul 08 19:37:17.515041 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.514986    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="449c2246-2433-40e4-91aa-89ee95528c53" containerName="istio-proxy"
Jul 08 19:37:17.518885 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.518851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.522607 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.522579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e3bca4-serving-cert\""
Jul 08 19:37:17.536458 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.535309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:17.562970 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.562936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"]
Jul 08 19:37:17.566470 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.566438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/97f39720-789d-4539-9ca4-4f9f03693388-kube-api-access-fcdzt\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.566826 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.566806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.566953 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.566937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f39720-789d-4539-9ca4-4f9f03693388-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.629289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.629256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:17.670189 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.670149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.670460 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.670437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f39720-789d-4539-9ca4-4f9f03693388-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.670675 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.670656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/97f39720-789d-4539-9ca4-4f9f03693388-kube-api-access-fcdzt\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.671244 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:17.671223    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-serving-cert: secret "maas-api-e2e-shared-b-e3bca4-serving-cert" not found
Jul 08 19:37:17.671422 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:37:17.671411    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls podName:97f39720-789d-4539-9ca4-4f9f03693388 nodeName:}" failed. No retries permitted until 2026-07-08 19:37:18.171387664 +0000 UTC m=+2296.815408374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls") pod "maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x" (UID: "97f39720-789d-4539-9ca4-4f9f03693388") : secret "maas-api-e2e-shared-b-e3bca4-serving-cert" not found
Jul 08 19:37:17.672429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.672407    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f39720-789d-4539-9ca4-4f9f03693388-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.683293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.683231    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/97f39720-789d-4539-9ca4-4f9f03693388-kube-api-access-fcdzt\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:17.761359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.761324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:17.863538 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:17.863501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:18.177461 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:18.177367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:18.179968 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:18.179940    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls\") pod \"maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:18.209777 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:18.209746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:18.313907 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:18.313866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:18.435154 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:18.435062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:18.663819 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:18.663308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"]
Jul 08 19:37:18.665830 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:37:18.665795    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f39720_789d_4539_9ca4_4f9f03693388.slice/crio-ccf96455f604db26ca15a65ca1f68c687f8ba8e5541049f961fe17ff61fb3d53 WatchSource:0}: Error finding container ccf96455f604db26ca15a65ca1f68c687f8ba8e5541049f961fe17ff61fb3d53: Status 404 returned error can't find the container with id ccf96455f604db26ca15a65ca1f68c687f8ba8e5541049f961fe17ff61fb3d53
Jul 08 19:37:19.498072 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:19.498033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x" event={"ID":"97f39720-789d-4539-9ca4-4f9f03693388","Type":"ContainerStarted","Data":"f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1"}
Jul 08 19:37:19.498543 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:19.498094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x" event={"ID":"97f39720-789d-4539-9ca4-4f9f03693388","Type":"ContainerStarted","Data":"ccf96455f604db26ca15a65ca1f68c687f8ba8e5541049f961fe17ff61fb3d53"}
Jul 08 19:37:19.498543 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:19.498272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:19.517325 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:19.517281    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x" podStartSLOduration=2.216427416 podStartE2EDuration="2.517262091s" totalImagesPullingTime="300.834675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:37:19.516379763 +0000 UTC m=+2298.160400507" watchObservedRunningTime="2026-07-08 19:37:19.517262091 +0000 UTC m=+2298.161282821"
Jul 08 19:37:21.915415 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:21.915372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:23.521268 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:23.521236    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerID="98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be" exitCode=0
Jul 08 19:37:23.521761 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:23.521314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" event={"ID":"e7143a01-12ff-4547-ac65-e355f2b555ef","Type":"ContainerDied","Data":"98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be"}
Jul 08 19:37:23.521823 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:23.521770    2563 scope.go:117] "RemoveContainer" containerID="98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be"
Jul 08 19:37:23.522832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:23.522806    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:23.522901 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:23.522876    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:23.522947 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:23.522910    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:24.432735 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:24.432698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:24.529805 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:24.529767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" event={"ID":"e7143a01-12ff-4547-ac65-e355f2b555ef","Type":"ContainerStarted","Data":"eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b"}
Jul 08 19:37:24.741844 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:24.741754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:24.743205 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:24.742718    2563 patch_prober.go:28] interesting pod/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 08 19:37:24.743205 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:24.742813    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 08 19:37:25.509267 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:25.509239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:37:25.741575 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:25.741521    2563 patch_prober.go:28] interesting pod/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 08 19:37:25.742095 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:25.741605    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 08 19:37:26.741570 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:26.741526    2563 patch_prober.go:28] interesting pod/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 08 19:37:26.741796 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:26.741599    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 08 19:37:27.742002 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:27.741959    2563 patch_prober.go:28] interesting pod/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 08 19:37:27.742488 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:27.742033    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 08 19:37:28.745237 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:28.745207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:28.745727 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:28.745649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:28.746446 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:28.746430    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:37:29.565396 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:29.565305    2563 generic.go:356] "Generic (PLEG): container finished" podID="09613d34-6c30-4753-b47f-da8de162a37d" containerID="4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02" exitCode=0
Jul 08 19:37:29.565396 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:29.565384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" event={"ID":"09613d34-6c30-4753-b47f-da8de162a37d","Type":"ContainerDied","Data":"4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02"}
Jul 08 19:37:29.566025 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:29.566000    2563 scope.go:117] "RemoveContainer" containerID="4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02"
Jul 08 19:37:29.567171 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:29.567146    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:29.567231 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:29.567218    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:29.567273 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:29.567252    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 08 19:37:30.572394 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:30.572342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" event={"ID":"09613d34-6c30-4753-b47f-da8de162a37d","Type":"ContainerStarted","Data":"11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9"}
Jul 08 19:37:30.813955 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:30.813919    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:30.814957 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:30.814931    2563 patch_prober.go:28] interesting pod/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 08 19:37:30.815084 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:30.814985    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 08 19:37:30.878602 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:30.878505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:30.979381 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:30.979345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:31.813697 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:31.813661    2563 patch_prober.go:28] interesting pod/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 08 19:37:31.814158 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:31.813725    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 08 19:37:32.484771 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:32.484740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:32.813417 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:32.813353    2563 patch_prober.go:28] interesting pod/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 08 19:37:32.813586 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:32.813439    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 08 19:37:33.813903 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:33.813866    2563 patch_prober.go:28] interesting pod/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Jul 08 19:37:33.814286 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:33.813921    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Jul 08 19:37:34.817054 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:34.817026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:34.818136 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:34.818088    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:37:39.072867 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:39.072836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:37:39.173253 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:37:39.173208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:39:02.417720 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:39:02.417679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:39:02.425079 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:39:02.425055    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:39:02.429744 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:39:02.429726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:39:02.436479 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:39:02.436463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-55wp8_304f1fb3-642c-4933-bc87-3db8e0afbefc/ovn-acl-logging/0.log"
Jul 08 19:39:35.076032 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:39:35.075993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:26.951587 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:26.951552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:27.379105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.379079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:27.476662 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.476602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:27.617200 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.617157    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"]
Jul 08 19:40:27.617458 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.617432    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c" podUID="c477f4be-db06-4d16-91b1-98e1671e3b65" containerName="maas-api" containerID="cri-o://1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602" gracePeriod=30
Jul 08 19:40:27.638110 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.638086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:27.870907 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.870884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:40:27.913212 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.913121    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c477f4be-db06-4d16-91b1-98e1671e3b65-openshift-service-ca\") pod \"c477f4be-db06-4d16-91b1-98e1671e3b65\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") "
Jul 08 19:40:27.913212 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.913202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls\") pod \"c477f4be-db06-4d16-91b1-98e1671e3b65\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") "
Jul 08 19:40:27.913449 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.913260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvrf9\" (UniqueName: \"kubernetes.io/projected/c477f4be-db06-4d16-91b1-98e1671e3b65-kube-api-access-xvrf9\") pod \"c477f4be-db06-4d16-91b1-98e1671e3b65\" (UID: \"c477f4be-db06-4d16-91b1-98e1671e3b65\") "
Jul 08 19:40:27.913564 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.913538    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c477f4be-db06-4d16-91b1-98e1671e3b65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c477f4be-db06-4d16-91b1-98e1671e3b65" (UID: "c477f4be-db06-4d16-91b1-98e1671e3b65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:40:27.915301 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.915266    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c477f4be-db06-4d16-91b1-98e1671e3b65" (UID: "c477f4be-db06-4d16-91b1-98e1671e3b65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:40:27.915421 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:27.915390    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c477f4be-db06-4d16-91b1-98e1671e3b65-kube-api-access-xvrf9" (OuterVolumeSpecName: "kube-api-access-xvrf9") pod "c477f4be-db06-4d16-91b1-98e1671e3b65" (UID: "c477f4be-db06-4d16-91b1-98e1671e3b65"). InnerVolumeSpecName "kube-api-access-xvrf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:40:28.013986 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.013948    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c477f4be-db06-4d16-91b1-98e1671e3b65-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:28.013986 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.013977    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvrf9\" (UniqueName: \"kubernetes.io/projected/c477f4be-db06-4d16-91b1-98e1671e3b65-kube-api-access-xvrf9\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:28.013986 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.013989    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c477f4be-db06-4d16-91b1-98e1671e3b65-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:28.522889 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.522855    2563 generic.go:356] "Generic (PLEG): container finished" podID="c477f4be-db06-4d16-91b1-98e1671e3b65" containerID="1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602" exitCode=0
Jul 08 19:40:28.523055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.522904    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"
Jul 08 19:40:28.523055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.522916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c" event={"ID":"c477f4be-db06-4d16-91b1-98e1671e3b65","Type":"ContainerDied","Data":"1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602"}
Jul 08 19:40:28.523055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.522976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c" event={"ID":"c477f4be-db06-4d16-91b1-98e1671e3b65","Type":"ContainerDied","Data":"a886c52e48600c9986aa82b10d2ea68103809263d5e4b47ec2f97ec0890ed9b6"}
Jul 08 19:40:28.523055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.522999    2563 scope.go:117] "RemoveContainer" containerID="1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602"
Jul 08 19:40:28.532188 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.532160    2563 scope.go:117] "RemoveContainer" containerID="1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602"
Jul 08 19:40:28.532467 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:40:28.532447    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602\": container with ID starting with 1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602 not found: ID does not exist" containerID="1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602"
Jul 08 19:40:28.532525 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.532477    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602"} err="failed to get container status \"1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602\": rpc error: code = NotFound desc = could not find container \"1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602\": container with ID starting with 1c4339fa6d3d1e7a63b8ce028419c0f4fac67030ffb6545a59c5f0bf6be45602 not found: ID does not exist"
Jul 08 19:40:28.541427 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.541400    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"]
Jul 08 19:40:28.543759 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:28.543740    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-866a5b-76f8c95486-flx4c"]
Jul 08 19:40:30.456570 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:30.456534    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c477f4be-db06-4d16-91b1-98e1671e3b65" path="/var/lib/kubelet/pods/c477f4be-db06-4d16-91b1-98e1671e3b65/volumes"
Jul 08 19:40:33.726261 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:33.726225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:33.825841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:33.825790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:33.989164 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:33.989140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"]
Jul 08 19:40:33.989364 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:33.989339    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy" containerID="cri-o://eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b" gracePeriod=30
Jul 08 19:40:34.014331 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:34.014285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:34.428755 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:34.428677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:34.814810 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:34.814742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:34.914880 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:34.914848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:35.016422 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.016389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:35.089585 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.089487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"]
Jul 08 19:40:35.089832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.089803    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x" podUID="97f39720-789d-4539-9ca4-4f9f03693388" containerName="maas-api" containerID="cri-o://f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1" gracePeriod=30
Jul 08 19:40:35.111479 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.111456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:35.339002 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.338980    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:40:35.383064 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.382979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/97f39720-789d-4539-9ca4-4f9f03693388-kube-api-access-fcdzt\") pod \"97f39720-789d-4539-9ca4-4f9f03693388\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") "
Jul 08 19:40:35.383064 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.383023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f39720-789d-4539-9ca4-4f9f03693388-openshift-service-ca\") pod \"97f39720-789d-4539-9ca4-4f9f03693388\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") "
Jul 08 19:40:35.383294 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.383072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls\") pod \"97f39720-789d-4539-9ca4-4f9f03693388\" (UID: \"97f39720-789d-4539-9ca4-4f9f03693388\") "
Jul 08 19:40:35.383416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.383393    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f39720-789d-4539-9ca4-4f9f03693388-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97f39720-789d-4539-9ca4-4f9f03693388" (UID: "97f39720-789d-4539-9ca4-4f9f03693388"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:40:35.383553 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.383534    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97f39720-789d-4539-9ca4-4f9f03693388-openshift-service-ca\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:35.385162 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.385135    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f39720-789d-4539-9ca4-4f9f03693388-kube-api-access-fcdzt" (OuterVolumeSpecName: "kube-api-access-fcdzt") pod "97f39720-789d-4539-9ca4-4f9f03693388" (UID: "97f39720-789d-4539-9ca4-4f9f03693388"). InnerVolumeSpecName "kube-api-access-fcdzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:40:35.385275 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.385190    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97f39720-789d-4539-9ca4-4f9f03693388" (UID: "97f39720-789d-4539-9ca4-4f9f03693388"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 19:40:35.484964 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.484935    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcdzt\" (UniqueName: \"kubernetes.io/projected/97f39720-789d-4539-9ca4-4f9f03693388-kube-api-access-fcdzt\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:35.484964 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.484960    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97f39720-789d-4539-9ca4-4f9f03693388-maas-api-tls\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:35.560348 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.560319    2563 generic.go:356] "Generic (PLEG): container finished" podID="97f39720-789d-4539-9ca4-4f9f03693388" containerID="f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1" exitCode=0
Jul 08 19:40:35.560500 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.560363    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"
Jul 08 19:40:35.560500 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.560382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x" event={"ID":"97f39720-789d-4539-9ca4-4f9f03693388","Type":"ContainerDied","Data":"f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1"}
Jul 08 19:40:35.560500 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.560426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x" event={"ID":"97f39720-789d-4539-9ca4-4f9f03693388","Type":"ContainerDied","Data":"ccf96455f604db26ca15a65ca1f68c687f8ba8e5541049f961fe17ff61fb3d53"}
Jul 08 19:40:35.560500 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.560455    2563 scope.go:117] "RemoveContainer" containerID="f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1"
Jul 08 19:40:35.572784 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.572763    2563 scope.go:117] "RemoveContainer" containerID="f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1"
Jul 08 19:40:35.573060 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:40:35.573039    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1\": container with ID starting with f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1 not found: ID does not exist" containerID="f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1"
Jul 08 19:40:35.573101 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.573069    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1"} err="failed to get container status \"f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1\": rpc error: code = NotFound desc = could not find container \"f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1\": container with ID starting with f81fbb4b8e518d72daa975c34e4fe9c4ddc364535284902592138f85aa9061b1 not found: ID does not exist"
Jul 08 19:40:35.584876 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.584851    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"]
Jul 08 19:40:35.591268 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:35.591244    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e3bca4-5cfbb8c685-hrq4x"]
Jul 08 19:40:36.455832 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:36.455802    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f39720-789d-4539-9ca4-4f9f03693388" path="/var/lib/kubelet/pods/97f39720-789d-4539-9ca4-4f9f03693388/volumes"
Jul 08 19:40:39.248161 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.248139    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:40:39.321919 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.321847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqcxb\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-kube-api-access-lqcxb\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.321919 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.321908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7143a01-12ff-4547-ac65-e355f2b555ef-istiod-ca-cert\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322110 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.321948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-podinfo\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322110 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-envoy\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322110 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-socket\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322110 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322084    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-token\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-data\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-certs\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322300 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-credential-socket\") pod \"e7143a01-12ff-4547-ac65-e355f2b555ef\" (UID: \"e7143a01-12ff-4547-ac65-e355f2b555ef\") "
Jul 08 19:40:39.322460 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322298    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7143a01-12ff-4547-ac65-e355f2b555ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:40:39.322460 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322391    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:39.322604 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322551    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:39.322826 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322555    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7143a01-12ff-4547-ac65-e355f2b555ef-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.322914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322848    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.322914 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.322599    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:39.323043 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.323021    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:39.324347 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.324326    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:40:39.324429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.324374    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:40:39.324525 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.324504    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:39.324574 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.324533    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-kube-api-access-lqcxb" (OuterVolumeSpecName: "kube-api-access-lqcxb") pod "e7143a01-12ff-4547-ac65-e355f2b555ef" (UID: "e7143a01-12ff-4547-ac65-e355f2b555ef"). InnerVolumeSpecName "kube-api-access-lqcxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:40:39.423979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.423947    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.423979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.423973    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.423979 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.423985    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.424201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.423997    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.424201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.424008    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqcxb\" (UniqueName: \"kubernetes.io/projected/e7143a01-12ff-4547-ac65-e355f2b555ef-kube-api-access-lqcxb\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.424201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.424020    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.424201 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.424030    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7143a01-12ff-4547-ac65-e355f2b555ef-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:39.584069 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.583976    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerID="eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b" exitCode=0
Jul 08 19:40:39.584069 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.584026    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"
Jul 08 19:40:39.584285 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.584059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" event={"ID":"e7143a01-12ff-4547-ac65-e355f2b555ef","Type":"ContainerDied","Data":"eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b"}
Jul 08 19:40:39.584285 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.584112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v" event={"ID":"e7143a01-12ff-4547-ac65-e355f2b555ef","Type":"ContainerDied","Data":"73cc1f4331c17cfd080e420ae3d16b650d25ca479104a8b50f6ac9f2ab3580cd"}
Jul 08 19:40:39.584285 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.584133    2563 scope.go:117] "RemoveContainer" containerID="eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b"
Jul 08 19:40:39.600185 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.600157    2563 scope.go:117] "RemoveContainer" containerID="98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be"
Jul 08 19:40:39.607959 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.607942    2563 scope.go:117] "RemoveContainer" containerID="eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b"
Jul 08 19:40:39.608215 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:40:39.608199    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b\": container with ID starting with eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b not found: ID does not exist" containerID="eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b"
Jul 08 19:40:39.608259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.608223    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b"} err="failed to get container status \"eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b\": rpc error: code = NotFound desc = could not find container \"eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b\": container with ID starting with eb047a3465f9f9a0e160254c9dfdac7f0edd5e13a6d200e9431f268a87f6050b not found: ID does not exist"
Jul 08 19:40:39.608259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.608239    2563 scope.go:117] "RemoveContainer" containerID="98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be"
Jul 08 19:40:39.608449 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:40:39.608435    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be\": container with ID starting with 98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be not found: ID does not exist" containerID="98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be"
Jul 08 19:40:39.608485 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.608452    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be"} err="failed to get container status \"98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be\": rpc error: code = NotFound desc = could not find container \"98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be\": container with ID starting with 98f2b678f72c3f04b4f56c61e5c447262967484124fb90f9141932e8f6c3e9be not found: ID does not exist"
Jul 08 19:40:39.612947 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.612926    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"]
Jul 08 19:40:39.617055 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:39.617034    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-866a5b-openshift-default-7f89c69795-s547v"]
Jul 08 19:40:40.456148 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:40.456117    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" path="/var/lib/kubelet/pods/e7143a01-12ff-4547-ac65-e355f2b555ef/volumes"
Jul 08 19:40:40.862225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:40.862193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:41.008352 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:41.008279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:41.030436 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:41.030407    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"]
Jul 08 19:40:41.030669 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:41.030642    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy" containerID="cri-o://11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9" gracePeriod=30
Jul 08 19:40:41.106562 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:41.106523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:46.282592 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.282569    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:40:46.387746 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387666    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-envoy\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387746 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387727    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09613d34-6c30-4753-b47f-da8de162a37d-istiod-ca-cert\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-certs\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387765    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-istio-token\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387781    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-data\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjq4c\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-kube-api-access-wjq4c\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09613d34-6c30-4753-b47f-da8de162a37d-istio-podinfo\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-credential-socket\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.387980 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.387963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-socket\") pod \"09613d34-6c30-4753-b47f-da8de162a37d\" (UID: \"09613d34-6c30-4753-b47f-da8de162a37d\") "
Jul 08 19:40:46.388359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.388028    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:46.388359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.388252    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:46.388359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.388180    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09613d34-6c30-4753-b47f-da8de162a37d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 19:40:46.388359 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.388309    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-certs\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.388568 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.388360    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:46.388857 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.388813    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-data" (OuterVolumeSpecName: "istio-data") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:46.390012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.389989    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-istio-token" (OuterVolumeSpecName: "istio-token") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:40:46.390012 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.390002    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 19:40:46.390149 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.390103    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09613d34-6c30-4753-b47f-da8de162a37d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 19:40:46.390259 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.390244    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-kube-api-access-wjq4c" (OuterVolumeSpecName: "kube-api-access-wjq4c") pod "09613d34-6c30-4753-b47f-da8de162a37d" (UID: "09613d34-6c30-4753-b47f-da8de162a37d"). InnerVolumeSpecName "kube-api-access-wjq4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 19:40:46.489741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489714    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-workload-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.489741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489737    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-envoy\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.489741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489747    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09613d34-6c30-4753-b47f-da8de162a37d-istiod-ca-cert\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.489930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489755    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-istio-token\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.489930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489762    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-istio-data\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.489930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489770    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjq4c\" (UniqueName: \"kubernetes.io/projected/09613d34-6c30-4753-b47f-da8de162a37d-kube-api-access-wjq4c\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.489930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489779    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09613d34-6c30-4753-b47f-da8de162a37d-istio-podinfo\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.489930 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.489788    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09613d34-6c30-4753-b47f-da8de162a37d-credential-socket\") on node \"ip-10-0-128-89.ec2.internal\" DevicePath \"\""
Jul 08 19:40:46.623536 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.623504    2563 generic.go:356] "Generic (PLEG): container finished" podID="09613d34-6c30-4753-b47f-da8de162a37d" containerID="11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9" exitCode=0
Jul 08 19:40:46.623702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.623556    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"
Jul 08 19:40:46.623702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.623596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" event={"ID":"09613d34-6c30-4753-b47f-da8de162a37d","Type":"ContainerDied","Data":"11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9"}
Jul 08 19:40:46.623702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.623669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n" event={"ID":"09613d34-6c30-4753-b47f-da8de162a37d","Type":"ContainerDied","Data":"37b5afb1245cb333fdcc210423fcbf0243a89950c6404571cd5039976ba127f3"}
Jul 08 19:40:46.623702 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.623692    2563 scope.go:117] "RemoveContainer" containerID="11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9"
Jul 08 19:40:46.632666 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.632618    2563 scope.go:117] "RemoveContainer" containerID="4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02"
Jul 08 19:40:46.640706 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.640685    2563 scope.go:117] "RemoveContainer" containerID="11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9"
Jul 08 19:40:46.640942 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:40:46.640920    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9\": container with ID starting with 11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9 not found: ID does not exist" containerID="11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9"
Jul 08 19:40:46.641019 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.640949    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9"} err="failed to get container status \"11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9\": rpc error: code = NotFound desc = could not find container \"11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9\": container with ID starting with 11c7529ab80221758371c52e0e2060c1e2559fe963303f2598fbab63f28cafa9 not found: ID does not exist"
Jul 08 19:40:46.641019 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.640969    2563 scope.go:117] "RemoveContainer" containerID="4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02"
Jul 08 19:40:46.641180 ip-10-0-128-89 kubenswrapper[2563]: E0708 19:40:46.641165    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02\": container with ID starting with 4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02 not found: ID does not exist" containerID="4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02"
Jul 08 19:40:46.641229 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.641183    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02"} err="failed to get container status \"4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02\": rpc error: code = NotFound desc = could not find container \"4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02\": container with ID starting with 4b4ee285ae3a67baf45fec3be91dfb48628c4a071a046df0b182907d9806cf02 not found: ID does not exist"
Jul 08 19:40:46.648739 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.648718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"]
Jul 08 19:40:46.653072 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.653045    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e3bca4-openshift-default-6c54678df5-wcm7n"]
Jul 08 19:40:46.712727 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.712703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7fcfb68c87-j72rp_8b33789c-a165-491d-8f9f-d9ceeabe6150/manager/0.log"
Jul 08 19:40:46.827601 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:46.827576    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xmdfl_03487c46-ee6c-4f8f-b703-4747db34c9af/server/0.log"
Jul 08 19:40:47.073888 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:47.073860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd7775959-2c4cn_05f2e1bb-a497-494f-a5e9-2d40189f287b/copy-manifests/0.log"
Jul 08 19:40:47.081773 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:47.081755    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd7775959-2c4cn_05f2e1bb-a497-494f-a5e9-2d40189f287b/manager/0.log"
Jul 08 19:40:48.142309 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.142284    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj_e78667e8-b469-4030-a5a4-16e3ba683528/util/0.log"
Jul 08 19:40:48.149179 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.149137    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj_e78667e8-b469-4030-a5a4-16e3ba683528/pull/0.log"
Jul 08 19:40:48.155293 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.155276    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj_e78667e8-b469-4030-a5a4-16e3ba683528/extract/0.log"
Jul 08 19:40:48.264831 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.264806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d_764cbca2-713e-46ae-ba62-be468308f6e3/util/0.log"
Jul 08 19:40:48.271530 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.271510    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d_764cbca2-713e-46ae-ba62-be468308f6e3/pull/0.log"
Jul 08 19:40:48.277584 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.277564    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d_764cbca2-713e-46ae-ba62-be468308f6e3/extract/0.log"
Jul 08 19:40:48.388189 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.388162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q_97ced5c2-b241-48ce-981f-3a93ad25e215/pull/0.log"
Jul 08 19:40:48.395028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.394938    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q_97ced5c2-b241-48ce-981f-3a93ad25e215/extract/0.log"
Jul 08 19:40:48.401855 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.401832    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q_97ced5c2-b241-48ce-981f-3a93ad25e215/util/0.log"
Jul 08 19:40:48.455607 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.455576    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09613d34-6c30-4753-b47f-da8de162a37d" path="/var/lib/kubelet/pods/09613d34-6c30-4753-b47f-da8de162a37d/volumes"
Jul 08 19:40:48.511231 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.511207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r_90afc36d-4dcd-48af-8676-8084cda5a3c1/pull/0.log"
Jul 08 19:40:48.517468 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.517447    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r_90afc36d-4dcd-48af-8676-8084cda5a3c1/extract/0.log"
Jul 08 19:40:48.523147 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.523130    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r_90afc36d-4dcd-48af-8676-8084cda5a3c1/util/0.log"
Jul 08 19:40:48.884221 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:48.884191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-z4qqc_318ba7b6-0cb9-401f-b981-2a3b7339003d/manager/0.log"
Jul 08 19:40:49.828320 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:49.828289    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v9lvc_d3c9f26c-cd40-43b9-bca0-daab0c641500/istio-proxy/0.log"
Jul 08 19:40:49.949351 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:49.949310    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnpm_9b33312d-7f53-452b-9489-1eb07e098525/discovery/0.log"
Jul 08 19:40:50.066269 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:50.066220    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bd9454df-ll5cn_22f64f61-b641-4fd8-a445-f5bafd042491/kube-auth-proxy/0.log"
Jul 08 19:40:50.300769 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:50.300732    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cdrm2_8dd6eb0d-e25c-4fb5-9a14-c85c50626e61/istio-proxy/1.log"
Jul 08 19:40:50.416861 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:50.416828    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-58t7x_f886fde6-1a29-4db8-8596-5f9ca1d94034/payload-pre-processing/0.log"
Jul 08 19:40:50.527225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:50.527185    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sm57r_5180632f-51cb-4843-a11d-76de044d5d8c/payload-processing/0.log"
Jul 08 19:40:50.993667 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:50.993617    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9rwb7_3a0bf640-b2e7-4cef-82ed-ba29425c1037/main/0.log"
Jul 08 19:40:51.111955 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:51.111915    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9rnwt_f3416c93-1c30-4f6a-b27e-ef432104bc4d/main/0.log"
Jul 08 19:40:51.349521 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:51.349433    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkm4mn_71be099e-7c18-4fd9-bdde-199d3f998c7a/main/0.log"
Jul 08 19:40:56.108034 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:56.108004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:56.208470 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:56.208409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log" limit=1024
Jul 08 19:40:59.202791 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:59.202762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mlmxj_e43d8623-63ae-4e5b-92e7-fb1db07f2cec/global-pull-secret-syncer/0.log"
Jul 08 19:40:59.251026 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:59.250998    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dls7k_df0230e9-590c-4a18-9dde-52eaa9fdb8ce/konnectivity-agent/0.log"
Jul 08 19:40:59.348170 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:40:59.348143    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-89.ec2.internal_db49d6c0b18dfe371c7b893a3266857e/haproxy/0.log"
Jul 08 19:41:03.322181 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.322135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj_e78667e8-b469-4030-a5a4-16e3ba683528/extract/0.log"
Jul 08 19:41:03.351499 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.351468    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj_e78667e8-b469-4030-a5a4-16e3ba683528/util/0.log"
Jul 08 19:41:03.376668 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.376619    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599xfmj_e78667e8-b469-4030-a5a4-16e3ba683528/pull/0.log"
Jul 08 19:41:03.423270 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.423247    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d_764cbca2-713e-46ae-ba62-be468308f6e3/extract/0.log"
Jul 08 19:41:03.442336 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.442316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d_764cbca2-713e-46ae-ba62-be468308f6e3/util/0.log"
Jul 08 19:41:03.465987 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.465957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xtw9d_764cbca2-713e-46ae-ba62-be468308f6e3/pull/0.log"
Jul 08 19:41:03.497376 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.497333    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q_97ced5c2-b241-48ce-981f-3a93ad25e215/extract/0.log"
Jul 08 19:41:03.517615 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.517590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q_97ced5c2-b241-48ce-981f-3a93ad25e215/util/0.log"
Jul 08 19:41:03.539667 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.539644    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c627q_97ced5c2-b241-48ce-981f-3a93ad25e215/pull/0.log"
Jul 08 19:41:03.572515 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.572461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r_90afc36d-4dcd-48af-8676-8084cda5a3c1/extract/0.log"
Jul 08 19:41:03.593416 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.593390    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r_90afc36d-4dcd-48af-8676-8084cda5a3c1/util/0.log"
Jul 08 19:41:03.620882 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:03.620865    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1djw6r_90afc36d-4dcd-48af-8676-8084cda5a3c1/pull/0.log"
Jul 08 19:41:04.234171 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:04.234143    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-z4qqc_318ba7b6-0cb9-401f-b981-2a3b7339003d/manager/0.log"
Jul 08 19:41:06.790741 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:06.790716    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjknm_cc9fda5a-6055-41a1-bf07-2e546728b400/node-exporter/0.log"
Jul 08 19:41:06.817838 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:06.817815    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjknm_cc9fda5a-6055-41a1-bf07-2e546728b400/kube-rbac-proxy/0.log"
Jul 08 19:41:06.845383 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:06.845363    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjknm_cc9fda5a-6055-41a1-bf07-2e546728b400/init-textfile/0.log"
Jul 08 19:41:09.208063 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.208035    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/2.log"
Jul 08 19:41:09.212587 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.212566    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-m929j_8d1cd19b-8af5-479c-8bbd-47402f98e466/console-operator/3.log"
Jul 08 19:41:09.327225 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327192    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"]
Jul 08 19:41:09.327719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327659    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy"
Jul 08 19:41:09.327719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327685    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy"
Jul 08 19:41:09.327719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327711    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy"
Jul 08 19:41:09.327719 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327719    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327728    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f39720-789d-4539-9ca4-4f9f03693388" containerName="maas-api"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327735    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f39720-789d-4539-9ca4-4f9f03693388" containerName="maas-api"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327752    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c477f4be-db06-4d16-91b1-98e1671e3b65" containerName="maas-api"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327759    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c477f4be-db06-4d16-91b1-98e1671e3b65" containerName="maas-api"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327776    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327784    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327879    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7143a01-12ff-4547-ac65-e355f2b555ef" containerName="istio-proxy"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327892    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f39720-789d-4539-9ca4-4f9f03693388" containerName="maas-api"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327901    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327907    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09613d34-6c30-4753-b47f-da8de162a37d" containerName="istio-proxy"
Jul 08 19:41:09.328105 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.327914    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c477f4be-db06-4d16-91b1-98e1671e3b65" containerName="maas-api"
Jul 08 19:41:09.330895 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.330880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.334082 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.334063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7j27\"/\"openshift-service-ca.crt\""
Jul 08 19:41:09.334176 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.334101    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7j27\"/\"kube-root-ca.crt\""
Jul 08 19:41:09.335427 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.335412    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c7j27\"/\"default-dockercfg-z67fq\""
Jul 08 19:41:09.341965 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.341940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"]
Jul 08 19:41:09.406440 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.406415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-podres\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.406597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.406513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-lib-modules\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.406597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.406537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-proc\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.406597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.406556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4f2m\" (UniqueName: \"kubernetes.io/projected/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-kube-api-access-m4f2m\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.406597 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.406576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-sys\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507525 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-lib-modules\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507675 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-proc\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507675 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4f2m\" (UniqueName: \"kubernetes.io/projected/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-kube-api-access-m4f2m\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507675 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-sys\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507675 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-podres\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507675 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507657    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-proc\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507717    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-lib-modules\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507757    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-sys\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.507878 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.507802    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-podres\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.516463 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.516446    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4f2m\" (UniqueName: \"kubernetes.io/projected/aabca4e2-0216-4d93-b102-1c7e1c9ac3f3-kube-api-access-m4f2m\") pod \"perf-node-gather-daemonset-clxs7\" (UID: \"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3\") " pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.642060 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.642032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:09.774649 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.774609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"]
Jul 08 19:41:09.775371 ip-10-0-128-89 kubenswrapper[2563]: W0708 19:41:09.775344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaabca4e2_0216_4d93_b102_1c7e1c9ac3f3.slice/crio-800702054b956b4c7dd56e03c0cf008892ec8537eea982fca1f16644b5b1941a WatchSource:0}: Error finding container 800702054b956b4c7dd56e03c0cf008892ec8537eea982fca1f16644b5b1941a: Status 404 returned error can't find the container with id 800702054b956b4c7dd56e03c0cf008892ec8537eea982fca1f16644b5b1941a
Jul 08 19:41:09.866619 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:09.866599    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7778bdfcc-v8llj_ca93bd6a-fd2f-494b-bb3c-6341f71efac9/console/0.log"
Jul 08 19:41:10.750911 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:10.750876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7" event={"ID":"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3","Type":"ContainerStarted","Data":"1501585034cd94d35dc0f11b6adddc02372e3fdd9decf7c5adc5b4eca40c65af"}
Jul 08 19:41:10.751283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:10.750933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7" event={"ID":"aabca4e2-0216-4d93-b102-1c7e1c9ac3f3","Type":"ContainerStarted","Data":"800702054b956b4c7dd56e03c0cf008892ec8537eea982fca1f16644b5b1941a"}
Jul 08 19:41:10.751283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:10.751010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:10.770232 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:10.770184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7" podStartSLOduration=1.770168349 podStartE2EDuration="1.770168349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:41:10.768206403 +0000 UTC m=+2529.412227134" watchObservedRunningTime="2026-07-08 19:41:10.770168349 +0000 UTC m=+2529.414189079"
Jul 08 19:41:11.539417 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:11.539393    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8rz2p_f21f6bbf-67fd-46e4-aff0-291624037c2c/dns/0.log"
Jul 08 19:41:11.562247 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:11.562223    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8rz2p_f21f6bbf-67fd-46e4-aff0-291624037c2c/kube-rbac-proxy/0.log"
Jul 08 19:41:11.701818 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:11.701776    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n4x4s_4fea7aea-3311-4b3a-bbf9-6a0c2585aec6/dns-node-resolver/0.log"
Jul 08 19:41:12.280637 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:12.280596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6fd4f585c9-5ftj8_297d68f5-6075-415b-b5bb-76a120365d16/registry/0.log"
Jul 08 19:41:12.299429 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:12.299409    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2ztqx_77bd5f3c-b7d3-4a92-b408-8366970336ef/node-ca/0.log"
Jul 08 19:41:13.267282 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:13.267248    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v9lvc_d3c9f26c-cd40-43b9-bca0-daab0c641500/istio-proxy/0.log"
Jul 08 19:41:13.368841 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:13.368810    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnpm_9b33312d-7f53-452b-9489-1eb07e098525/discovery/0.log"
Jul 08 19:41:13.390097 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:13.390075    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bd9454df-ll5cn_22f64f61-b641-4fd8-a445-f5bafd042491/kube-auth-proxy/0.log"
Jul 08 19:41:13.442029 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:13.441999    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cdrm2_8dd6eb0d-e25c-4fb5-9a14-c85c50626e61/istio-proxy/0.log"
Jul 08 19:41:13.468111 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:13.468084    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cdrm2_8dd6eb0d-e25c-4fb5-9a14-c85c50626e61/istio-proxy/1.log"
Jul 08 19:41:13.506028 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:13.506005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-58t7x_f886fde6-1a29-4db8-8596-5f9ca1d94034/payload-pre-processing/0.log"
Jul 08 19:41:13.550765 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:13.550687    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sm57r_5180632f-51cb-4843-a11d-76de044d5d8c/payload-processing/0.log"
Jul 08 19:41:14.249532 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:14.249500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn77p_488e72ac-a322-4fb9-8553-936a24cc6683/serve-healthcheck-canary/0.log"
Jul 08 19:41:14.944974 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:14.944949    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrzk6_70e1619c-bd2b-40ef-a1d5-91954ea233dd/kube-rbac-proxy/0.log"
Jul 08 19:41:14.966198 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:14.966176    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrzk6_70e1619c-bd2b-40ef-a1d5-91954ea233dd/exporter/0.log"
Jul 08 19:41:14.987335 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:14.987309    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrzk6_70e1619c-bd2b-40ef-a1d5-91954ea233dd/extractor/0.log"
Jul 08 19:41:16.766292 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:16.766266    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c7j27/perf-node-gather-daemonset-clxs7"
Jul 08 19:41:17.294543 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:17.294496    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7fcfb68c87-j72rp_8b33789c-a165-491d-8f9f-d9ceeabe6150/manager/0.log"
Jul 08 19:41:17.318658 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:17.318618    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xmdfl_03487c46-ee6c-4f8f-b703-4747db34c9af/server/0.log"
Jul 08 19:41:17.449898 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:17.449869    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd7775959-2c4cn_05f2e1bb-a497-494f-a5e9-2d40189f287b/manager/0.log"
Jul 08 19:41:17.474764 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:17.474736    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd7775959-2c4cn_05f2e1bb-a497-494f-a5e9-2d40189f287b/copy-manifests/0.log"
Jul 08 19:41:19.686211 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:19.686182    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v9lvc_d3c9f26c-cd40-43b9-bca0-daab0c641500/istio-proxy/0.log"
Jul 08 19:41:19.774182 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:19.774153    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnpm_9b33312d-7f53-452b-9489-1eb07e098525/discovery/0.log"
Jul 08 19:41:19.784717 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:19.784695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bd9454df-ll5cn_22f64f61-b641-4fd8-a445-f5bafd042491/kube-auth-proxy/0.log"
Jul 08 19:41:19.819703 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:19.819676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cdrm2_8dd6eb0d-e25c-4fb5-9a14-c85c50626e61/istio-proxy/0.log"
Jul 08 19:41:19.846948 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:19.846924    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cdrm2_8dd6eb0d-e25c-4fb5-9a14-c85c50626e61/istio-proxy/1.log"
Jul 08 19:41:19.876380 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:19.876358    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-58t7x_f886fde6-1a29-4db8-8596-5f9ca1d94034/payload-pre-processing/0.log"
Jul 08 19:41:19.919219 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:19.919195    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sm57r_5180632f-51cb-4843-a11d-76de044d5d8c/payload-processing/0.log"
Jul 08 19:41:21.197897 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:21.197808    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v9lvc_d3c9f26c-cd40-43b9-bca0-daab0c641500/istio-proxy/0.log"
Jul 08 19:41:21.293289 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:21.293263    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnpm_9b33312d-7f53-452b-9489-1eb07e098525/discovery/0.log"
Jul 08 19:41:21.306906 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:21.306882    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85bd9454df-ll5cn_22f64f61-b641-4fd8-a445-f5bafd042491/kube-auth-proxy/0.log"
Jul 08 19:41:21.347269 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:21.347248    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cdrm2_8dd6eb0d-e25c-4fb5-9a14-c85c50626e61/istio-proxy/0.log"
Jul 08 19:41:21.374447 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:21.374419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cdrm2_8dd6eb0d-e25c-4fb5-9a14-c85c50626e61/istio-proxy/1.log"
Jul 08 19:41:21.403549 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:21.403524    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-58t7x_f886fde6-1a29-4db8-8596-5f9ca1d94034/payload-pre-processing/0.log"
Jul 08 19:41:21.442565 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:21.442540    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sm57r_5180632f-51cb-4843-a11d-76de044d5d8c/payload-processing/0.log"
Jul 08 19:41:25.483344 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:25.483267    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5m9cq_43f058dd-d85e-4c3c-9212-3e7095606619/kube-multus-additional-cni-plugins/0.log"
Jul 08 19:41:25.504614 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:25.504588    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5m9cq_43f058dd-d85e-4c3c-9212-3e7095606619/egress-router-binary-copy/0.log"
Jul 08 19:41:25.529588 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:25.529564    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5m9cq_43f058dd-d85e-4c3c-9212-3e7095606619/cni-plugins/0.log"
Jul 08 19:41:25.552455 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:25.552434    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5m9cq_43f058dd-d85e-4c3c-9212-3e7095606619/bond-cni-plugin/0.log"
Jul 08 19:41:25.579382 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:25.579362    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5m9cq_43f058dd-d85e-4c3c-9212-3e7095606619/routeoverride-cni/0.log"
Jul 08 19:41:25.601283 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:25.601261    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5m9cq_43f058dd-d85e-4c3c-9212-3e7095606619/whereabouts-cni-bincopy/0.log"
Jul 08 19:41:25.622595 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:25.622568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5m9cq_43f058dd-d85e-4c3c-9212-3e7095606619/whereabouts-cni/0.log"
Jul 08 19:41:26.057447 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:26.057414    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zqgfd_7c15e4dd-dbca-407c-85b7-079bc169c852/kube-multus/0.log"
Jul 08 19:41:26.173211 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:26.173184    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r6pt5_daca3b10-16da-4559-8498-a0bfec82aad8/network-metrics-daemon/0.log"
Jul 08 19:41:26.203406 ip-10-0-128-89 kubenswrapper[2563]: I0708 19:41:26.203383    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r6pt5_daca3b10-16da-4559-8498-a0bfec82aad8/kube-rbac-proxy/0.log"