Jul 07 01:01:11.620966 ip-10-0-142-16 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 01:01:12.023167 ip-10-0-142-16 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 01:01:12.023167 ip-10-0-142-16 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 01:01:12.023167 ip-10-0-142-16 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 01:01:12.023167 ip-10-0-142-16 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 01:01:12.023167 ip-10-0-142-16 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 01:01:12.025266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.025202    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 01:01:12.030544 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030520    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 01:01:12.030544 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030541    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 01:01:12.030544 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030544    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030551    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030554    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030557    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030561    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030563    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030567    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030570    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030573    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030576    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030579    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030581    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030586    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030593    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030596    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030599    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030602    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030604    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030607    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030610    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 01:01:12.030667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030612    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030617    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030621    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030624    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030630    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030633    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030636    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030639    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030642    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030645    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030647    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030650    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030653    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030656    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030659    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030662    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030665    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030670    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030674    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 01:01:12.031144 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030676    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030697    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030701    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030703    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030706    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030708    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030711    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030714    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030716    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030719    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030724    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030727    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030729    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030733    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030736    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030738    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030741    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030743    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030746    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030748    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 01:01:12.031604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030751    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030755    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030760    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030763    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030766    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030768    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030771    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030775    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030778    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030781    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030783    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030786    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030789    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030791    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030794    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030799    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030801    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030803    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030806    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 01:01:12.032143 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030808    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030811    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030814    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030816    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030818    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030821    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030823    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030826    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030831    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030833    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030836    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030838    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030841    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030843    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030846    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030848    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030851    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030853    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.030856    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032403    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 01:01:12.032597 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032411    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032414    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032416    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032419    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032422    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032425    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032428    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032430    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032433    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032435    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032441    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032443    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032446    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032449    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032453    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032455    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032458    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032460    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032463    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 01:01:12.033081 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032466    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032468    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032472    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032478    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032481    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032484    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032487    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032490    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032492    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032495    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032497    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032500    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032502    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032505    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032508    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032510    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032516    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032518    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032521    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 01:01:12.033539 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032524    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032527    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032531    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032533    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032536    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032539    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032541    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032544    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032547    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032552    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032555    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032558    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032561    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032563    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032566    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032568    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032570    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032573    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032575    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032578    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 01:01:12.034007 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032580    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032585    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032588    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032590    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032594    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032598    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032601    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032604    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032608    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032612    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032616    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032619    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032626    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032630    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032633    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032636    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032639    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032642    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032645    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 01:01:12.034491 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032648    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032651    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032654    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032656    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032659    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032662    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032666    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032669    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032672    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032675    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032695    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032698    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032701    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032704    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032706    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032709    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032711    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032714    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032716    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032721    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 01:01:12.034955 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.032724    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033304    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033313    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033316    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033330    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033333    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033336    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033339    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033342    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033345    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033348    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033350    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033353    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033359    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033362    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033365    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033368    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033371    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033373    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033376    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 01:01:12.035435 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033378    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033380    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033383    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033386    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033388    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033393    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033396    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033399    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033401    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033404    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033406    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033409    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033411    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033414    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033418    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033420    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033423    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033426    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033431    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033434    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 01:01:12.035935 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033436    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033439    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033441    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033444    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033446    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033449    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033451    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033454    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033457    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033460    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033465    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033467    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033470    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033472    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033475    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033478    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033480    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033483    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033485    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033488    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 01:01:12.036448 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033491    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033493    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033498    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033502    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033505    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033507    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033511    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033516    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033520    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033523    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033526    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033528    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033531    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033537    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033539    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033542    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033545    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033547    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033550    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 01:01:12.036966 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033552    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033556    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033559    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033562    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033565    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033568    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033571    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033576    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033578    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033581    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033584    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033587    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033589    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033592    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033595    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033598    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033600    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033603    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033605    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 01:01:12.037429 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033611    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.033613    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033796    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033809    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033903    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033921    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033939    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033944    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033952    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033960    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033964    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033967    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033970    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033973    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033976    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033979    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033981    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033984    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033987    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033990    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033996    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.033998    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034001    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034006    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 01:01:12.037892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034011    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034018    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034023    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034027    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034030    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034033    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034036    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034041    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034044    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034047    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034051    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034054    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034057    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034060    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034062    2562 flags.go:64] FLAG: --enable-server="true"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034065    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034071    2562 flags.go:64] FLAG: --event-burst="100"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034074    2562 flags.go:64] FLAG: --event-qps="50"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034077    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034080    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034082    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034086    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034089    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034093    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034098    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 07 01:01:12.038438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034101    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034103    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034108    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034111    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034115    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034117    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034121    2562 flags.go:64] FLAG: --feature-gates=""
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034125    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034128    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034131    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034133    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034137    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034139    2562 flags.go:64] FLAG: --help="false"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034142    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034146    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034149    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034152    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034155    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034160    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034163    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034165    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034168    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034171    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 01:01:12.039050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034174    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034177    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034180    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034183    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034186    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034188    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034191    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034194    2562 flags.go:64] FLAG: --lock-file=""
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034196    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034199    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034202    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034212    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034216    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034218    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034221    2562 flags.go:64] FLAG: --logging-format="text"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034224    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034227    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034230    2562 flags.go:64] FLAG: --manifest-url=""
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034232    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034237    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034239    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034243    2562 flags.go:64] FLAG: --max-pods="110"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034246    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034250    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034253    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 01:01:12.039637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034255    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034258    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034263    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034265    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034273    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034276    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034279    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034282    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034284    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034289    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034292    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034295    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034297    2562 flags.go:64] FLAG: --port="10250"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034300    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034303    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-019ce8fcb043e2f14"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034306    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034309    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034312    2562 flags.go:64] FLAG: --register-node="true"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034315    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034318    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034321    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034324    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034326    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034330    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034333    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 01:01:12.040290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034336    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034338    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034341    2562 flags.go:64] FLAG: --runonce="false"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034344    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034347    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034349    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034353    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034356    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034359    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034363    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034366    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034369    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034371    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034374    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034377    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034379    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034382    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034384    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034390    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034392    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034395    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034399    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034402    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034404    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034407    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 01:01:12.041073 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034410    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034412    2562 flags.go:64] FLAG: --v="2"
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034416    2562 flags.go:64] FLAG: --version="false"
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034420    2562 flags.go:64] FLAG: --vmodule=""
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034423    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.034426    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034560    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034564    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034567    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034570    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034572    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034575    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034578    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034582    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034584    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034587    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034591    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034593    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034596    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034598    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034601    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 01:01:12.041668 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034603    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034606    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034608    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034611    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034613    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034616    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034619    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034621    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034623    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034626    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034628    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034631    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034633    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034637    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034652    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034656    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034659    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034662    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034665    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034668    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 01:01:12.042197 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034670    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034673    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034676    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034694    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034698    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034700    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034703    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034707    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034709    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034712    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034714    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034718    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034721    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034724    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034727    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034729    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034731    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034734    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034736    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034739    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 01:01:12.042677 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034741    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034744    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034746    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034749    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034752    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034754    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034757    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034759    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034762    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034764    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034767    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034769    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034773    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034776    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034779    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034781    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034785    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034787    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034790    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034793    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 01:01:12.043186 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034796    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034799    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034801    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034804    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034806    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034809    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034811    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034814    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034816    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034818    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034821    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034823    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034826    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034828    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034830    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034833    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034835    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034838    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034840    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034843    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 01:01:12.043706 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034845    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034848    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034851    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.034853    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.035601    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.042291    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.042303    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.043204    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.043214    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043267    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043271    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043275    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043278    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043280    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043284    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 01:01:12.044183 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043287    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043291    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043294    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043297    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043300    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043302    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043304    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043307    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043309    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043312    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043315    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043317    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043319    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043322    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043324    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043326    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043329    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043331    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043334    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043336    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 01:01:12.044547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043338    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043341    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043343    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043346    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043349    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043351    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043354    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043356    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043358    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043361    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043363    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043366    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043368    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043371    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043373    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043376    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043379    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043383    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043385    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 01:01:12.045123 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043388    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043391    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043393    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043396    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043399    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043401    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043404    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043407    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043409    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043412    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043415    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043417    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043420    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043422    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043424    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043427    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043429    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043432    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043434    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043437    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 01:01:12.045609 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043440    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043442    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043444    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043446    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043449    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043452    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043454    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043457    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043459    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043461    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043464    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043467    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043469    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043472    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043475    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043478    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043481    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043484    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043486    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 01:01:12.046109 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043489    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043492    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043494    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043497    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043499    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043501    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043504    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043506    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043509    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043511    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043513    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043517    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043519    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043521    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043530    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 01:01:12.046564 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.043536    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043639    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043644    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043647    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043650    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043653    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043655    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043658    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043661    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043663    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043665    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043668    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043670    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043673    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043675    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043677    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043694    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043697    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043699    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 01:01:12.046952 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043702    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043704    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043707    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043709    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043712    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043714    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043717    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043720    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043722    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043725    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043728    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043730    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043732    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043735    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043737    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043740    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043743    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043746    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043748    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043751    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 01:01:12.047398 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043753    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043755    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043758    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043761    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043763    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043766    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043768    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043771    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043773    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043776    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043778    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043780    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043783    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043785    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043787    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043790    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043792    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043794    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043797    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043799    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 01:01:12.047902 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043801    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043804    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043806    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043810    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043812    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043814    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043817    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043819    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043821    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043824    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043827    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043829    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043831    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043833    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043836    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043838    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043841    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043843    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043845    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043848    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 01:01:12.048379 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043850    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043852    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043855    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043858    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043860    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043862    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043865    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043867    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043871    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043874    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043877    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043880    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043882    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043884    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043887    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043889    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043892    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043894    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043898    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 01:01:12.048900 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043902    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 01:01:12.049351 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:12.043905    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 01:01:12.049351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.043911    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 01:01:12.049351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.044002    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 01:01:12.049612 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.049598    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 01:01:12.051340 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.051329    2562 server.go:1019] "Starting client certificate rotation"
Jul 07 01:01:12.051441 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.051427    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 01:01:12.051496 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.051488    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 01:01:12.054119 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.054107    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 01:01:12.055177 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.055153    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 01:01:12.068781 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.068760    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 07 01:01:12.072774 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.072759    2562 log.go:25] "Validated CRI v1 image API"
Jul 07 01:01:12.074854 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.074838    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 01:01:12.077544 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.077527    2562 fs.go:135] Filesystem UUIDs: map[5c249c9d-7c59-486e-9919-23b1f2ad0b23:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 faa53e42-4645-4201-9d61-3f229b8aa787:/dev/nvme0n1p3]
Jul 07 01:01:12.077605 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.077544    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 01:01:12.084063 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.083962    2562 manager.go:216] Machine: {Timestamp:2026-07-07 01:01:12.082518012 +0000 UTC m=+0.357509325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099590 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac4112a2fd3aa6a44664773a62000 SystemUUID:ec2ac411-2a2f-d3aa-6a44-664773a62000 BootID:467291a7-2619-4855-a1dd-1cd79ffbf04f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b4:f2:34:9f:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b4:f2:34:9f:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:2c:6f:cf:75:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 01:01:12.084063 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.084055    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 01:01:12.084167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.084122    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 01:01:12.086003 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.085987    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 01:01:12.087986 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.087960    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 01:01:12.088132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.087991    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 01:01:12.089703 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.089693    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 01:01:12.089739 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.089706    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 01:01:12.089739 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.089713    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 01:01:12.090485 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.090476    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 01:01:12.092365 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.092354    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 01:01:12.097001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.096990    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 01:01:12.097228 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097219    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 01:01:12.097264 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097249    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 01:01:12.097291 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097266    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 01:01:12.097291 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097283    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 01:01:12.097343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097292    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 07 01:01:12.097343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097300    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 01:01:12.097603 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097592    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 01:01:12.097634 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.097611    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 01:01:12.099859 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.099844    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 01:01:12.101990 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.101977    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 01:01:12.102033 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.102007    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 01:01:12.103467 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103456    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 01:01:12.103513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103473    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 01:01:12.103513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103479    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 01:01:12.103513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103485    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 01:01:12.103513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103501    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 01:01:12.103513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103507    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 01:01:12.103513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103514    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 01:01:12.103700 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103520    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 01:01:12.103700 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103535    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 01:01:12.103700 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103540    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 01:01:12.103700 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103553    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 01:01:12.103700 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.103558    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 01:01:12.105249 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.105239    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 01:01:12.105249 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.105248    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 01:01:12.107851 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.107834    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xqvgm"
Jul 07 01:01:12.108219 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.108206    2562 server.go:1268] "Started kubelet"
Jul 07 01:01:12.108304 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.108281    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 01:01:12.108443 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.108328    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 01:01:12.108583 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.108570    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 01:01:12.109031 ip-10-0-142-16 systemd[1]: Started Kubernetes Kubelet.
Jul 07 01:01:12.110083 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.109859    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 01:01:12.110083 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.110050    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 01:01:12.110382 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.110361    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 01:01:12.111038 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.111016    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 01:01:12.111729 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.111717    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 01:01:12.115648 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.115624    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xqvgm"
Jul 07 01:01:12.116202 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.116178    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 01:01:12.118930 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.117848    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-16.ec2.internal.18bfdc2f0d632b32  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-16.ec2.internal,UID:ip-10-0-142-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-16.ec2.internal,},FirstTimestamp:2026-07-07 01:01:12.108182322 +0000 UTC m=+0.383173637,LastTimestamp:2026-07-07 01:01:12.108182322 +0000 UTC m=+0.383173637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-16.ec2.internal,}"
Jul 07 01:01:12.127373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.127360    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 01:01:12.127452 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.127368    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 01:01:12.127452 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.127433    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.128469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128450    2562 factory.go:55] Registering systemd factory
Jul 07 01:01:12.128469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128472    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 07 01:01:12.128607 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128533    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 01:01:12.128607 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128547    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 01:01:12.128607 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128560    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 01:01:12.128733 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128672    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 01:01:12.128733 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128693    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 01:01:12.128733 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128715    2562 factory.go:153] Registering CRI-O factory
Jul 07 01:01:12.128733 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128730    2562 factory.go:223] Registration of the crio container factory successfully
Jul 07 01:01:12.128842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128784    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 01:01:12.128842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128805    2562 factory.go:103] Registering Raw factory
Jul 07 01:01:12.128842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.128821    2562 manager.go:1196] Started watching for new ooms in manager
Jul 07 01:01:12.129136 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.129125    2562 manager.go:349] Starting recovery of all containers
Jul 07 01:01:12.130301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.130279    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 01:01:12.133174 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.133151    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-16.ec2.internal\" not found" node="ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.138966 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.138862    2562 manager.go:354] Recovery completed
Jul 07 01:01:12.142899 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.142884    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 01:01:12.144498 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.144484    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 01:01:12.144558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.144512    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 01:01:12.144558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.144521    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID"
Jul 07 01:01:12.144972 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.144958    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 01:01:12.145082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.144973    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 01:01:12.145082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.144971    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 01:01:12.145082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.145031    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 01:01:12.146071 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.146056    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 01:01:12.146133 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.146083    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 01:01:12.146133 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.146093    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID"
Jul 07 01:01:12.147086 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.147075    2562 policy_none.go:49] "None policy: Start"
Jul 07 01:01:12.147126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.147090    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 01:01:12.147126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.147099    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 01:01:12.149053 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.149040    2562 policy_none.go:47] "Start"
Jul 07 01:01:12.190239 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.190225    2562 manager.go:337] "Starting Device Plugin manager"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.190252    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.190261    2562 server.go:87] "Starting device plugin registration server"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.190447    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.190458    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.190562    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.190648    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.190668    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.191070    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 01:01:12.211010 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.191120    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.281743 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.281671    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 01:01:12.282930 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.282917    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 01:01:12.282983 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.282950    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 01:01:12.282983 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.282965    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 01:01:12.283073 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.282992    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 01:01:12.285417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.285398    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 01:01:12.291360 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.291342    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 01:01:12.293486 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.293472    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 01:01:12.293550 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.293498    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 01:01:12.293550 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.293507    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID"
Jul 07 01:01:12.293550 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.293525    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.301361 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.301348    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.301407 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.301366    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-16.ec2.internal\": node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.314337 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.314314    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.383733 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.383708    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"]
Jul 07 01:01:12.383803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.383771    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 01:01:12.384503 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.384489    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 01:01:12.384568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.384513    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 01:01:12.384568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.384524    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID"
Jul 07 01:01:12.386864 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.386852    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 01:01:12.386981 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.386968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.387023 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.386996    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 01:01:12.387536 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.387512    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 01:01:12.387616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.387545    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 01:01:12.387616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.387560    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID"
Jul 07 01:01:12.387616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.387586    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 01:01:12.387616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.387614    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 01:01:12.387829 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.387628    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID"
Jul 07 01:01:12.389904 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.389887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.389991 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.389934    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 01:01:12.390587 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.390572    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 01:01:12.390665 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.390600    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 01:01:12.390665 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.390610    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeHasSufficientPID"
Jul 07 01:01:12.415232 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.415196    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.415386 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.415372    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-16.ec2.internal\" not found" node="ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.419746 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.419730    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-16.ec2.internal\" not found" node="ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.429932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.429896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dbb58ca2bc7089e0dfff1cca9c75a0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"9dbb58ca2bc7089e0dfff1cca9c75a0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.430002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.429940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a927f5a1e989d40b4e926ec92d449add-config\") pod \"kube-apiserver-proxy-ip-10-0-142-16.ec2.internal\" (UID: \"a927f5a1e989d40b4e926ec92d449add\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.430002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.429956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dbb58ca2bc7089e0dfff1cca9c75a0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"9dbb58ca2bc7089e0dfff1cca9c75a0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.516240 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.516208    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.530549 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.530529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dbb58ca2bc7089e0dfff1cca9c75a0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"9dbb58ca2bc7089e0dfff1cca9c75a0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.530600 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.530552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dbb58ca2bc7089e0dfff1cca9c75a0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"9dbb58ca2bc7089e0dfff1cca9c75a0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.530600 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.530568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a927f5a1e989d40b4e926ec92d449add-config\") pod \"kube-apiserver-proxy-ip-10-0-142-16.ec2.internal\" (UID: \"a927f5a1e989d40b4e926ec92d449add\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.530666 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.530603    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a927f5a1e989d40b4e926ec92d449add-config\") pod \"kube-apiserver-proxy-ip-10-0-142-16.ec2.internal\" (UID: \"a927f5a1e989d40b4e926ec92d449add\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.530666 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.530642    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dbb58ca2bc7089e0dfff1cca9c75a0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"9dbb58ca2bc7089e0dfff1cca9c75a0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.530748 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.530647    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dbb58ca2bc7089e0dfff1cca9c75a0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal\" (UID: \"9dbb58ca2bc7089e0dfff1cca9c75a0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.616918 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.616862    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.717405 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.717379    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.718494 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.718478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.723416 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:12.723268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:12.818385 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.818361    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:12.918957 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:12.918906    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:13.019491 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:13.019466    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:13.050937 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.050921    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 01:01:13.051408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.051051    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 01:01:13.051408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.051089    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 01:01:13.120149 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:13.120129    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:13.123863 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.123840    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 00:56:12 +0000 UTC" deadline="2028-03-25 07:40:48.231804896 +0000 UTC"
Jul 07 01:01:13.123906 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.123863    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15054h39m35.107944302s"
Jul 07 01:01:13.221159 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:13.221140    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-16.ec2.internal\" not found"
Jul 07 01:01:13.241504 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.241346    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 01:01:13.315101 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:13.315070    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda927f5a1e989d40b4e926ec92d449add.slice/crio-9bbc2b4fb0fd078f76e65f982488d02d466140f38cde40c0f332499c7ce895b8 WatchSource:0}: Error finding container 9bbc2b4fb0fd078f76e65f982488d02d466140f38cde40c0f332499c7ce895b8: Status 404 returned error can't find the container with id 9bbc2b4fb0fd078f76e65f982488d02d466140f38cde40c0f332499c7ce895b8
Jul 07 01:01:13.318610 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.318597    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 01:01:13.318863 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.318848    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 01:01:13.328478 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.328462    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:13.338231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.338215    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 01:01:13.339238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.339226    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal"
Jul 07 01:01:13.346835 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.346822    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 01:01:13.592889 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:13.592869    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbb58ca2bc7089e0dfff1cca9c75a0d.slice/crio-c4a9454982a20c52f3fa0ce94eab82ae7b52b6e0f9b49d843fe47f49bc644bbc WatchSource:0}: Error finding container c4a9454982a20c52f3fa0ce94eab82ae7b52b6e0f9b49d843fe47f49bc644bbc: Status 404 returned error can't find the container with id c4a9454982a20c52f3fa0ce94eab82ae7b52b6e0f9b49d843fe47f49bc644bbc
Jul 07 01:01:13.901197 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:13.901134    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 01:01:14.013780 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.013756    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 01:01:14.099126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.099107    2562 apiserver.go:52] "Watching apiserver"
Jul 07 01:01:14.108107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.108081    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 01:01:14.108528 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.108500    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ddjh7","kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz","openshift-cluster-node-tuning-operator/tuned-dd47n","openshift-dns/node-resolver-75844","openshift-image-registry/node-ca-97djw","openshift-multus/network-metrics-daemon-gf5c2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal","openshift-multus/multus-additional-cni-plugins-2fkb2","openshift-multus/multus-v2ml8","openshift-network-diagnostics/network-check-target-gx5ms","openshift-network-operator/iptables-alerter-b7mcs","openshift-ovn-kubernetes/ovnkube-node-t4h9k","kube-system/global-pull-secret-syncer-lqst9"]
Jul 07 01:01:14.111284 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.111266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:14.111362 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.111335    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:14.113434 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.113415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.115793 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.115766    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.116279 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.116254    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 01:01:14.116508 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.116489    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 01:01:14.116562 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.116537    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 01:01:14.117268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.117246    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 01:01:14.117268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.117257    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2g4f6\""
Jul 07 01:01:14.117417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.117256    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 01:01:14.117417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.117281    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 01:01:14.118101 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.118081    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.118469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.118442    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 01:01:14.118567 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.118545    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 01:01:14.118622 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.118591    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 01:01:14.118714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.118677    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kdfmt\""
Jul 07 01:01:14.120243 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.120216    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6n5b9\""
Jul 07 01:01:14.120243 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.120218    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.120444 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.120406    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 01:01:14.120511 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.120460    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 01:01:14.122417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.122394    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 01:01:14.122530 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.122465    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqxqc\""
Jul 07 01:01:14.122592 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.122528    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.122662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.122643    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 01:01:14.124876 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.124851    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 01:01:14.125020 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.124941    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 01:01:14.125119 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.125088    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8728w\""
Jul 07 01:01:14.125267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.125241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.125898 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.125880    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 01:01:14.127404 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.127385    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 01:01:14.127634 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.127614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.127740 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.127703    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 01:01:14.127740 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.127716    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 01:01:14.127853 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.127766    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5sb7l\""
Jul 07 01:01:14.127902 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.127882    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 01:01:14.127978 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.127956    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 01:01:14.128193 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.128168    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 01:01:14.130384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.130364    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7nfhw\""
Jul 07 01:01:14.130503 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.130396    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 01:01:14.130893 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.130865    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 01:01:14.131156 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.131125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.131255 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.131225    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:14.134566 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.134545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.136790 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.136699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:14.136790 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.136766    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:14.137027 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.137003    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 01:01:14.137095 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.137071    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ggvzt\""
Jul 07 01:01:14.138057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-tuned\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.138155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.138155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmv26\" (UniqueName: \"kubernetes.io/projected/8216316d-d40e-4cf4-8d4c-13974c5e5f74-kube-api-access-nmv26\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.138307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-device-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.138307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-host\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7sh4\" (UniqueName: \"kubernetes.io/projected/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-kube-api-access-q7sh4\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-run-netns\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.138307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85ba5407-90de-4401-a47c-51227f2e1958-dbus\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46bdeae8-a38b-4939-adbf-91f694cad4ca-agent-certs\") pod \"konnectivity-agent-ddjh7\" (UID: \"46bdeae8-a38b-4939-adbf-91f694cad4ca\") " pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46bdeae8-a38b-4939-adbf-91f694cad4ca-konnectivity-ca\") pod \"konnectivity-agent-ddjh7\" (UID: \"46bdeae8-a38b-4939-adbf-91f694cad4ca\") " pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovnkube-script-lib\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268x9\" (UniqueName: \"kubernetes.io/projected/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-kube-api-access-268x9\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysctl-conf\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-slash\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovn-node-metrics-cert\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-socket-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-lib-modules\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-ovn\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2d23b4d-300b-4264-b083-6172651d2548-host\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtz9s\" (UniqueName: \"kubernetes.io/projected/b2d23b4d-300b-4264-b083-6172651d2548-kube-api-access-rtz9s\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-kubelet\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-etc-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-registration-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n26h4\" (UniqueName: \"kubernetes.io/projected/c8f50afc-7a89-485b-a262-52c1aa1ca181-kube-api-access-n26h4\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-modprobe-d\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-sys\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cnibin\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-var-lib-kubelet\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-tmp\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovnkube-config\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85ba5407-90de-4401-a47c-51227f2e1958-kubelet-config\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-kubernetes\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.138965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2d23b4d-300b-4264-b083-6172651d2548-serviceca\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138915    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-cni-bin\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-log-socket\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa786802-732e-45a6-9420-7cfe2571fd09-tmp-dir\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-etc-selinux\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.138998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-run-ovn-kubernetes\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-system-cni-dir\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-os-release\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa786802-732e-45a6-9420-7cfe2571fd09-hosts-file\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysconfig\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysctl-d\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-systemd\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-run\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-node-log\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-cni-netd\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-env-overrides\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2m9s\" (UniqueName: \"kubernetes.io/projected/aa786802-732e-45a6-9420-7cfe2571fd09-kube-api-access-b2m9s\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-sys-fs\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-systemd-units\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-systemd\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139328    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-var-lib-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.139994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.139369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kmm\" (UniqueName: \"kubernetes.io/projected/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-kube-api-access-x2kmm\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:14.142280 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.142260    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 01:01:14.142382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.142261    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nc6tv\""
Jul 07 01:01:14.142382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.142299    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 01:01:14.142382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.142318    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 01:01:14.143282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.143189    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 01:01:14.163115 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.163100    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r7v8x"
Jul 07 01:01:14.169898 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.169878    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r7v8x"
Jul 07 01:01:14.229558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.229533    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 01:01:14.239632 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-etc-selinux\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.239738 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-run-ovn-kubernetes\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.239738 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-system-cni-dir\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.239738 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-os-release\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-cnibin\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239750    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-run-ovn-kubernetes\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-os-release\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239772    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-etc-selinux\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa786802-732e-45a6-9420-7cfe2571fd09-hosts-file\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239815    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-system-cni-dir\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239880    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-os-release\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-cni-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.239932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-multus-certs\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.239963    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa786802-732e-45a6-9420-7cfe2571fd09-hosts-file\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-etc-kubernetes\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysconfig\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysctl-d\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-systemd\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-run\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-node-log\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240136    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysconfig\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-cni-netd\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-env-overrides\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240171    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-node-log\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240180    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-systemd\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2m9s\" (UniqueName: \"kubernetes.io/projected/aa786802-732e-45a6-9420-7cfe2571fd09-kube-api-access-b2m9s\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240204    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-run\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240205    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-cni-netd\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-sys-fs\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.240282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240245    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysctl-d\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-systemd-units\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-systemd\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-var-lib-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240405    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-systemd\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240372    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-sys-fs\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240381    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-systemd-units\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kmm\" (UniqueName: \"kubernetes.io/projected/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-kube-api-access-x2kmm\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240428    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-var-lib-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwd5x\" (UniqueName: \"kubernetes.io/projected/a30b5907-4cbd-4f42-b751-0c6c0df427cc-kube-api-access-xwd5x\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcdj\" (UniqueName: \"kubernetes.io/projected/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-kube-api-access-krcdj\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-tuned\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240717    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240758    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-env-overrides\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.240885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmv26\" (UniqueName: \"kubernetes.io/projected/8216316d-d40e-4cf4-8d4c-13974c5e5f74-kube-api-access-nmv26\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-device-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-host\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240930    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-device-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sh4\" (UniqueName: \"kubernetes.io/projected/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-kube-api-access-q7sh4\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.240980    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241011    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-host\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-run-netns\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241053    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-run-netns\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85ba5407-90de-4401-a47c-51227f2e1958-dbus\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a30b5907-4cbd-4f42-b751-0c6c0df427cc-cni-binary-copy\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46bdeae8-a38b-4939-adbf-91f694cad4ca-agent-certs\") pod \"konnectivity-agent-ddjh7\" (UID: \"46bdeae8-a38b-4939-adbf-91f694cad4ca\") " pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46bdeae8-a38b-4939-adbf-91f694cad4ca-konnectivity-ca\") pod \"konnectivity-agent-ddjh7\" (UID: \"46bdeae8-a38b-4939-adbf-91f694cad4ca\") " pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovnkube-script-lib\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241275    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.241629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268x9\" (UniqueName: \"kubernetes.io/projected/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-kube-api-access-268x9\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241299    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-cni-multus\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysctl-conf\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241362    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241399    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85ba5407-90de-4401-a47c-51227f2e1958-dbus\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-slash\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovn-node-metrics-cert\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-k8s-cni-cncf-io\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241538    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-sysctl-conf\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-conf-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-socket-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241602    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-slash\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-lib-modules\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-ovn\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-kubelet\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-daemon-config\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.242390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241787    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-iptables-alerter-script\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2d23b4d-300b-4264-b083-6172651d2548-host\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241815    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-lib-modules\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241829    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtz9s\" (UniqueName: \"kubernetes.io/projected/b2d23b4d-300b-4264-b083-6172651d2548-kube-api-access-rtz9s\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241917    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46bdeae8-a38b-4939-adbf-91f694cad4ca-konnectivity-ca\") pod \"konnectivity-agent-ddjh7\" (UID: \"46bdeae8-a38b-4939-adbf-91f694cad4ca\") " pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.241981    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovnkube-script-lib\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242029    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-run-ovn\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242037    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2d23b4d-300b-4264-b083-6172651d2548-host\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-kubelet\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242113    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-socket-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242157    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-kubelet\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242209    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-socket-dir-parent\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-cni-bin\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-host-slash\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.243257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-etc-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242394    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-etc-openvswitch\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-registration-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n26h4\" (UniqueName: \"kubernetes.io/projected/c8f50afc-7a89-485b-a262-52c1aa1ca181-kube-api-access-n26h4\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242616    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-registration-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-modprobe-d\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-sys\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cnibin\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242774    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-modprobe-d\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242803    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-sys\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242839    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8216316d-d40e-4cf4-8d4c-13974c5e5f74-cnibin\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-var-lib-kubelet\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-tmp\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.242927    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.242919    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-var-lib-kubelet\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244091 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.243546    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs podName:2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:14.743478702 +0000 UTC m=+3.018470122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs") pod "network-metrics-daemon-gf5c2" (UID: "2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:14.244807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.244577    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-tuned\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.244807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.244780    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovn-node-metrics-cert\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.245049 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.245031    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46bdeae8-a38b-4939-adbf-91f694cad4ca-agent-certs\") pod \"konnectivity-agent-ddjh7\" (UID: \"46bdeae8-a38b-4939-adbf-91f694cad4ca\") " pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.246806 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.246779    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovnkube-config\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.246903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.246834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85ba5407-90de-4401-a47c-51227f2e1958-kubelet-config\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.246955 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.246890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-hostroot\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.246955 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.246937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-kubernetes\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.247045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.246969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.247045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.246996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-system-cni-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.247045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2d23b4d-300b-4264-b083-6172651d2548-serviceca\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.247192 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-cni-bin\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.247243 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-netns\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.247289 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-log-socket\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.247349 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247328    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8f50afc-7a89-485b-a262-52c1aa1ca181-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.247400 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa786802-732e-45a6-9420-7cfe2571fd09-tmp-dir\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247570    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-host-cni-bin\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.247635    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.247710    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret podName:85ba5407-90de-4401-a47c-51227f2e1958 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:14.747673752 +0000 UTC m=+3.022665068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret") pod "global-pull-secret-syncer-lqst9" (UID: "85ba5407-90de-4401-a47c-51227f2e1958") : object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247813    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-log-socket\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.247988    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-ovnkube-config\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.248059    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2d23b4d-300b-4264-b083-6172651d2548-serviceca\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.248064    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85ba5407-90de-4401-a47c-51227f2e1958-kubelet-config\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.248108    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa786802-732e-45a6-9420-7cfe2571fd09-tmp-dir\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.248789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.248186    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-etc-kubernetes\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.250810 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.250769    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7sh4\" (UniqueName: \"kubernetes.io/projected/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-kube-api-access-q7sh4\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.250906 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.250839    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2m9s\" (UniqueName: \"kubernetes.io/projected/aa786802-732e-45a6-9420-7cfe2571fd09-kube-api-access-b2m9s\") pod \"node-resolver-75844\" (UID: \"aa786802-732e-45a6-9420-7cfe2571fd09\") " pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.251828 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.251770    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kmm\" (UniqueName: \"kubernetes.io/projected/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-kube-api-access-x2kmm\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:14.252043 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.251990    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d71ae208-8fe0-4a84-a358-99f7e8c5b05b-tmp\") pod \"tuned-dd47n\" (UID: \"d71ae208-8fe0-4a84-a358-99f7e8c5b05b\") " pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.252043 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.252034    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtz9s\" (UniqueName: \"kubernetes.io/projected/b2d23b4d-300b-4264-b083-6172651d2548-kube-api-access-rtz9s\") pod \"node-ca-97djw\" (UID: \"b2d23b4d-300b-4264-b083-6172651d2548\") " pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.252229 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.252190    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n26h4\" (UniqueName: \"kubernetes.io/projected/c8f50afc-7a89-485b-a262-52c1aa1ca181-kube-api-access-n26h4\") pod \"aws-ebs-csi-driver-node-8vqbz\" (UID: \"c8f50afc-7a89-485b-a262-52c1aa1ca181\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.253324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.253305    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmv26\" (UniqueName: \"kubernetes.io/projected/8216316d-d40e-4cf4-8d4c-13974c5e5f74-kube-api-access-nmv26\") pod \"multus-additional-cni-plugins-2fkb2\" (UID: \"8216316d-d40e-4cf4-8d4c-13974c5e5f74\") " pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.253462 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.253446    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-268x9\" (UniqueName: \"kubernetes.io/projected/d61cb9e3-fa2c-44da-880f-3ba4a2892b47-kube-api-access-268x9\") pod \"ovnkube-node-t4h9k\" (UID: \"d61cb9e3-fa2c-44da-880f-3ba4a2892b47\") " pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.287645 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.287608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" event={"ID":"a927f5a1e989d40b4e926ec92d449add","Type":"ContainerStarted","Data":"9bbc2b4fb0fd078f76e65f982488d02d466140f38cde40c0f332499c7ce895b8"}
Jul 07 01:01:14.288687 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.288668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" event={"ID":"9dbb58ca2bc7089e0dfff1cca9c75a0d","Type":"ContainerStarted","Data":"c4a9454982a20c52f3fa0ce94eab82ae7b52b6e0f9b49d843fe47f49bc644bbc"}
Jul 07 01:01:14.348617 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwd5x\" (UniqueName: \"kubernetes.io/projected/a30b5907-4cbd-4f42-b751-0c6c0df427cc-kube-api-access-xwd5x\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348617 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcdj\" (UniqueName: \"kubernetes.io/projected/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-kube-api-access-krcdj\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.348783 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a30b5907-4cbd-4f42-b751-0c6c0df427cc-cni-binary-copy\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348783 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-cni-multus\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348783 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348731    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-cni-multus\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348926 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-k8s-cni-cncf-io\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348926 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-conf-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348926 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348855    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-k8s-cni-cncf-io\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348926 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-kubelet\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348926 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-daemon-config\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.348926 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348904    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-conf-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-iptables-alerter-script\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-socket-dir-parent\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348954    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-kubelet\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.348975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-cni-bin\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349043    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-var-lib-cni-bin\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349074    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-socket-dir-parent\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-host-slash\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-hostroot\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-system-cni-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-netns\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349198    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-host-slash\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349222    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-hostroot\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-cnibin\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349269    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-system-cni-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349276    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-netns\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349301    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-cnibin\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-os-release\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349319    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a30b5907-4cbd-4f42-b751-0c6c0df427cc-cni-binary-copy\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-cni-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349353    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-os-release\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-multus-certs\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349389    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-cni-dir\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349387    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a30b5907-4cbd-4f42-b751-0c6c0df427cc-multus-daemon-config\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349402    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-etc-kubernetes\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349421    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-host-run-multus-certs\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349455    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a30b5907-4cbd-4f42-b751-0c6c0df427cc-etc-kubernetes\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.349633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.349529    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-iptables-alerter-script\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.355410 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.355392    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 01:01:14.355410 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.355409    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 01:01:14.355551 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.355418    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gk8gr for pod openshift-network-diagnostics/network-check-target-gx5ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:14.355551 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.355472    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr podName:b57b8b8c-50df-48ce-83bb-7e5d175c3d8d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:14.855459838 +0000 UTC m=+3.130451141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gk8gr" (UniqueName: "kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr") pod "network-check-target-gx5ms" (UID: "b57b8b8c-50df-48ce-83bb-7e5d175c3d8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:14.357639 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.357613    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwd5x\" (UniqueName: \"kubernetes.io/projected/a30b5907-4cbd-4f42-b751-0c6c0df427cc-kube-api-access-xwd5x\") pod \"multus-v2ml8\" (UID: \"a30b5907-4cbd-4f42-b751-0c6c0df427cc\") " pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.357713 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.357655    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcdj\" (UniqueName: \"kubernetes.io/projected/06aa83ec-04af-4f19-a0d3-b17a6cc90ba3-kube-api-access-krcdj\") pod \"iptables-alerter-b7mcs\" (UID: \"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3\") " pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.427566 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.427521    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:14.434237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.434222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz"
Jul 07 01:01:14.439413 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.439387    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61cb9e3_fa2c_44da_880f_3ba4a2892b47.slice/crio-496d56a66459d163bb8418b4e5949b3a3714e6faa879a4ff678750fc7bd7ead3 WatchSource:0}: Error finding container 496d56a66459d163bb8418b4e5949b3a3714e6faa879a4ff678750fc7bd7ead3: Status 404 returned error can't find the container with id 496d56a66459d163bb8418b4e5949b3a3714e6faa879a4ff678750fc7bd7ead3
Jul 07 01:01:14.440949 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.440922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dd47n"
Jul 07 01:01:14.443999 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.443846    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f50afc_7a89_485b_a262_52c1aa1ca181.slice/crio-111a9a427b818ea2649d81e2f8cd1f617efcae39bcd63176ab46c47f7e21c3cc WatchSource:0}: Error finding container 111a9a427b818ea2649d81e2f8cd1f617efcae39bcd63176ab46c47f7e21c3cc: Status 404 returned error can't find the container with id 111a9a427b818ea2649d81e2f8cd1f617efcae39bcd63176ab46c47f7e21c3cc
Jul 07 01:01:14.446822 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.446797    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-75844"
Jul 07 01:01:14.451726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.451708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-97djw"
Jul 07 01:01:14.452075 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.452054    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71ae208_8fe0_4a84_a358_99f7e8c5b05b.slice/crio-a19dc51714bc488dddd7e03a5004acd8033ff08b6249e0aec7de59f4a6481669 WatchSource:0}: Error finding container a19dc51714bc488dddd7e03a5004acd8033ff08b6249e0aec7de59f4a6481669: Status 404 returned error can't find the container with id a19dc51714bc488dddd7e03a5004acd8033ff08b6249e0aec7de59f4a6481669
Jul 07 01:01:14.457997 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.457977    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa786802_732e_45a6_9420_7cfe2571fd09.slice/crio-c66712e452c50f8cde7a8628d5b64e43c49051a19151ddd629289ffcd87969ef WatchSource:0}: Error finding container c66712e452c50f8cde7a8628d5b64e43c49051a19151ddd629289ffcd87969ef: Status 404 returned error can't find the container with id c66712e452c50f8cde7a8628d5b64e43c49051a19151ddd629289ffcd87969ef
Jul 07 01:01:14.458081 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.458044    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2fkb2"
Jul 07 01:01:14.461135 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.460965    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d23b4d_300b_4264_b083_6172651d2548.slice/crio-d1381e3ba59476202237d4dd793e71df1f93484fc9c3c76008df94f1f969aa6d WatchSource:0}: Error finding container d1381e3ba59476202237d4dd793e71df1f93484fc9c3c76008df94f1f969aa6d: Status 404 returned error can't find the container with id d1381e3ba59476202237d4dd793e71df1f93484fc9c3c76008df94f1f969aa6d
Jul 07 01:01:14.465124 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.465107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:14.467024 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.467002    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8216316d_d40e_4cf4_8d4c_13974c5e5f74.slice/crio-ff3604d740f74a90d97cacdb226a760453aad75d81acde418028007ab4b24d8a WatchSource:0}: Error finding container ff3604d740f74a90d97cacdb226a760453aad75d81acde418028007ab4b24d8a: Status 404 returned error can't find the container with id ff3604d740f74a90d97cacdb226a760453aad75d81acde418028007ab4b24d8a
Jul 07 01:01:14.471562 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.470515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v2ml8"
Jul 07 01:01:14.475259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.475240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7mcs"
Jul 07 01:01:14.477112 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.477093    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bdeae8_a38b_4939_adbf_91f694cad4ca.slice/crio-f56ad05a83c6530f5b96eea9c26c44297fd267823da493ef0ea5bc35c98dff8d WatchSource:0}: Error finding container f56ad05a83c6530f5b96eea9c26c44297fd267823da493ef0ea5bc35c98dff8d: Status 404 returned error can't find the container with id f56ad05a83c6530f5b96eea9c26c44297fd267823da493ef0ea5bc35c98dff8d
Jul 07 01:01:14.480624 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.480594    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30b5907_4cbd_4f42_b751_0c6c0df427cc.slice/crio-b2ef546e72d440426a89eb28719a00ecb2bf7576b2bbb50a3f21f813f97977ad WatchSource:0}: Error finding container b2ef546e72d440426a89eb28719a00ecb2bf7576b2bbb50a3f21f813f97977ad: Status 404 returned error can't find the container with id b2ef546e72d440426a89eb28719a00ecb2bf7576b2bbb50a3f21f813f97977ad
Jul 07 01:01:14.485492 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:14.485472    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06aa83ec_04af_4f19_a0d3_b17a6cc90ba3.slice/crio-b7f7a1010b89cbad8b1687e284491f58c4e6e815ce044c13a7b472f7933cd8c5 WatchSource:0}: Error finding container b7f7a1010b89cbad8b1687e284491f58c4e6e815ce044c13a7b472f7933cd8c5: Status 404 returned error can't find the container with id b7f7a1010b89cbad8b1687e284491f58c4e6e815ce044c13a7b472f7933cd8c5
Jul 07 01:01:14.751459 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.751436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:14.751551 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.751467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:14.751606 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.751578    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:14.751606 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.751578    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:14.751670 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.751622    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret podName:85ba5407-90de-4401-a47c-51227f2e1958 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:15.751609474 +0000 UTC m=+4.026600775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret") pod "global-pull-secret-syncer-lqst9" (UID: "85ba5407-90de-4401-a47c-51227f2e1958") : object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:14.751670 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.751636    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs podName:2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:15.751629016 +0000 UTC m=+4.026620318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs") pod "network-metrics-daemon-gf5c2" (UID: "2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:14.952287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:14.952257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:14.952465 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.952442    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 01:01:14.952563 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.952470    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 01:01:14.952563 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.952484    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gk8gr for pod openshift-network-diagnostics/network-check-target-gx5ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:14.952563 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:14.952543    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr podName:b57b8b8c-50df-48ce-83bb-7e5d175c3d8d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:15.9525247 +0000 UTC m=+4.227516009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gk8gr" (UniqueName: "kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr") pod "network-check-target-gx5ms" (UID: "b57b8b8c-50df-48ce-83bb-7e5d175c3d8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:15.173458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.173370    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 00:56:14 +0000 UTC" deadline="2028-03-13 03:28:45.796521658 +0000 UTC"
Jul 07 01:01:15.173458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.173414    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14762h27m30.623112851s"
Jul 07 01:01:15.283584 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.283550    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:15.283751 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.283688    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:15.303166 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.303095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7mcs" event={"ID":"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3","Type":"ContainerStarted","Data":"b7f7a1010b89cbad8b1687e284491f58c4e6e815ce044c13a7b472f7933cd8c5"}
Jul 07 01:01:15.304728 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.304704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ddjh7" event={"ID":"46bdeae8-a38b-4939-adbf-91f694cad4ca","Type":"ContainerStarted","Data":"f56ad05a83c6530f5b96eea9c26c44297fd267823da493ef0ea5bc35c98dff8d"}
Jul 07 01:01:15.306822 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.306797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dd47n" event={"ID":"d71ae208-8fe0-4a84-a358-99f7e8c5b05b","Type":"ContainerStarted","Data":"a19dc51714bc488dddd7e03a5004acd8033ff08b6249e0aec7de59f4a6481669"}
Jul 07 01:01:15.308868 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.308841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"496d56a66459d163bb8418b4e5949b3a3714e6faa879a4ff678750fc7bd7ead3"}
Jul 07 01:01:15.311884 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.311858    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2ml8" event={"ID":"a30b5907-4cbd-4f42-b751-0c6c0df427cc","Type":"ContainerStarted","Data":"b2ef546e72d440426a89eb28719a00ecb2bf7576b2bbb50a3f21f813f97977ad"}
Jul 07 01:01:15.319838 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.319608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerStarted","Data":"ff3604d740f74a90d97cacdb226a760453aad75d81acde418028007ab4b24d8a"}
Jul 07 01:01:15.325911 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.325882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-97djw" event={"ID":"b2d23b4d-300b-4264-b083-6172651d2548","Type":"ContainerStarted","Data":"d1381e3ba59476202237d4dd793e71df1f93484fc9c3c76008df94f1f969aa6d"}
Jul 07 01:01:15.331812 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.331786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-75844" event={"ID":"aa786802-732e-45a6-9420-7cfe2571fd09","Type":"ContainerStarted","Data":"c66712e452c50f8cde7a8628d5b64e43c49051a19151ddd629289ffcd87969ef"}
Jul 07 01:01:15.335188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.335165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz" event={"ID":"c8f50afc-7a89-485b-a262-52c1aa1ca181","Type":"ContainerStarted","Data":"111a9a427b818ea2649d81e2f8cd1f617efcae39bcd63176ab46c47f7e21c3cc"}
Jul 07 01:01:15.759411 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.759367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:15.759593 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.759438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:15.759660 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.759593    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:15.759728 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.759660    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret podName:85ba5407-90de-4401-a47c-51227f2e1958 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:17.759639526 +0000 UTC m=+6.034630837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret") pod "global-pull-secret-syncer-lqst9" (UID: "85ba5407-90de-4401-a47c-51227f2e1958") : object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:15.760148 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.760127    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:15.760221 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.760208    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs podName:2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:17.760172906 +0000 UTC m=+6.035164218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs") pod "network-metrics-daemon-gf5c2" (UID: "2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:15.961982 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:15.961940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:15.962199 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.962162    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 01:01:15.962199 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.962186    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 01:01:15.962342 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.962213    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gk8gr for pod openshift-network-diagnostics/network-check-target-gx5ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:15.962342 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:15.962311    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr podName:b57b8b8c-50df-48ce-83bb-7e5d175c3d8d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:17.962260397 +0000 UTC m=+6.237251705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gk8gr" (UniqueName: "kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr") pod "network-check-target-gx5ms" (UID: "b57b8b8c-50df-48ce-83bb-7e5d175c3d8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:16.174597 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:16.174497    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 00:56:14 +0000 UTC" deadline="2028-03-19 15:54:05.43131483 +0000 UTC"
Jul 07 01:01:16.174597 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:16.174536    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14918h52m49.256783129s"
Jul 07 01:01:16.286646 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:16.286609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:16.286813 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:16.286781    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:16.287198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:16.287177    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:16.287328 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:16.287306    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:17.285501 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:17.284912    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:17.285501 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.285192    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:17.784759 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:17.784367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:17.784759 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:17.784461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:17.784759 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.784612    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:17.784759 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.784613    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:17.784759 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.784673    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs podName:2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:21.784652527 +0000 UTC m=+10.059643836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs") pod "network-metrics-daemon-gf5c2" (UID: "2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:17.784759 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.784693    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret podName:85ba5407-90de-4401-a47c-51227f2e1958 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:21.784682241 +0000 UTC m=+10.059673551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret") pod "global-pull-secret-syncer-lqst9" (UID: "85ba5407-90de-4401-a47c-51227f2e1958") : object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:17.986194 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:17.986135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:17.986376 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.986304    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 01:01:17.986376 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.986333    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 01:01:17.986376 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.986347    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gk8gr for pod openshift-network-diagnostics/network-check-target-gx5ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:17.986570 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:17.986423    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr podName:b57b8b8c-50df-48ce-83bb-7e5d175c3d8d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:21.986386044 +0000 UTC m=+10.261377353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gk8gr" (UniqueName: "kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr") pod "network-check-target-gx5ms" (UID: "b57b8b8c-50df-48ce-83bb-7e5d175c3d8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:18.284883 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:18.284846    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:18.285072 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:18.284985    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:18.285072 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:18.285050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:18.285223 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:18.285166    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:19.284082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:19.283533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:19.284082 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:19.283673    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:20.285062 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:20.284783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:20.285062 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:20.284903    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:20.285062 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:20.284903    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:20.285062 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:20.284990    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:21.283893 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:21.283864    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:21.284061 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:21.283986    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:21.814731 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:21.814696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:21.815201 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:21.814753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:21.815201 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:21.814812    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:21.815201 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:21.814883    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:21.815201 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:21.814890    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs podName:2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:29.814871079 +0000 UTC m=+18.089862394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs") pod "network-metrics-daemon-gf5c2" (UID: "2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:21.815201 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:21.814921    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret podName:85ba5407-90de-4401-a47c-51227f2e1958 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:29.814908929 +0000 UTC m=+18.089900234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret") pod "global-pull-secret-syncer-lqst9" (UID: "85ba5407-90de-4401-a47c-51227f2e1958") : object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:22.015726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:22.015696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:22.015868 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:22.015855    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 01:01:22.015904 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:22.015873    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 01:01:22.015904 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:22.015881    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gk8gr for pod openshift-network-diagnostics/network-check-target-gx5ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:22.015966 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:22.015921    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr podName:b57b8b8c-50df-48ce-83bb-7e5d175c3d8d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:30.015907376 +0000 UTC m=+18.290898677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gk8gr" (UniqueName: "kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr") pod "network-check-target-gx5ms" (UID: "b57b8b8c-50df-48ce-83bb-7e5d175c3d8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:22.287444 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:22.286762    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:22.287444 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:22.286904    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:22.287444 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:22.286971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:22.287444 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:22.287037    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:23.285101 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:23.285067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:23.285449 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:23.285175    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:24.286004 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:24.285971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:24.286414 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:24.286098    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:24.286969 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:24.286943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:24.287085 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:24.287030    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:25.283394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:25.283309    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:25.283576 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:25.283453    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:26.283841 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:26.283297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:26.283841 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:26.283418    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:26.283841 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:26.283297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:26.283841 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:26.283801    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:27.284167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:27.284134    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:27.284556 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:27.284228    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:28.284146 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:28.284114    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:28.284315 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:28.284233    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:28.284315 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:28.284289    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:28.284730 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:28.284399    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:29.283670 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:29.283636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:29.283943 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:29.283779    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:29.869724 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:29.869673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:29.870220 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:29.869745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:29.870220 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:29.869836    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:29.870220 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:29.869852    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:29.870220 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:29.869914    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs podName:2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:45.869892439 +0000 UTC m=+34.144883753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs") pod "network-metrics-daemon-gf5c2" (UID: "2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:29.870220 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:29.869935    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret podName:85ba5407-90de-4401-a47c-51227f2e1958 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:45.869926009 +0000 UTC m=+34.144917315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret") pod "global-pull-secret-syncer-lqst9" (UID: "85ba5407-90de-4401-a47c-51227f2e1958") : object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:30.071349 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:30.071315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:30.071521 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:30.071501    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 01:01:30.071573 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:30.071529    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 01:01:30.071573 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:30.071542    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gk8gr for pod openshift-network-diagnostics/network-check-target-gx5ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:30.071634 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:30.071606    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr podName:b57b8b8c-50df-48ce-83bb-7e5d175c3d8d nodeName:}" failed. No retries permitted until 2026-07-07 01:01:46.07158545 +0000 UTC m=+34.346576760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gk8gr" (UniqueName: "kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr") pod "network-check-target-gx5ms" (UID: "b57b8b8c-50df-48ce-83bb-7e5d175c3d8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:30.284227 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:30.284197    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:30.284409 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:30.284199    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:30.284409 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:30.284329    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:30.284521 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:30.284403    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:31.284150 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:31.284115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:31.284620 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:31.284236    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:32.284350 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:32.284314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:32.284813 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:32.284411    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:32.284813 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:32.284472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:32.284813 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:32.284577    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:33.284029 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:33.283997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:33.284190 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:33.284109    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:34.283968 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:34.283925    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:34.284375 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:34.283931    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:34.284375 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:34.284067    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:34.284375 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:34.284115    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:35.283929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:35.283888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:35.284209 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:35.284007    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:36.283635 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.283425    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:36.283867 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.283303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:36.283984 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:36.283945    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:36.284108 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:36.284088    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:36.391514 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.391484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" event={"ID":"a927f5a1e989d40b4e926ec92d449add","Type":"ContainerStarted","Data":"98c3b3b2068d52fb6086bb1864573b72ea2f185cebdb7b723391e06401cf8cbf"}
Jul 07 01:01:36.393188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.393161    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v2ml8" event={"ID":"a30b5907-4cbd-4f42-b751-0c6c0df427cc","Type":"ContainerStarted","Data":"8a9eaa536cd5668354ce3ec3856d04d67e26ae2ac4f8c3042150149666ec030a"}
Jul 07 01:01:36.394729 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.394701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dd47n" event={"ID":"d71ae208-8fe0-4a84-a358-99f7e8c5b05b","Type":"ContainerStarted","Data":"e14e1d52cc8c18cefcf386020c17d1bc2283e9e5f6fcc249450cb4785fb4c963"}
Jul 07 01:01:36.398371 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398341    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:01:36.398746 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398725    2562 generic.go:356] "Generic (PLEG): container finished" podID="d61cb9e3-fa2c-44da-880f-3ba4a2892b47" containerID="0e9d558c61e5b002956e1337d1c1c3ad801dcdfc766a5644d8c4ff7678064673" exitCode=1
Jul 07 01:01:36.398811 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398756    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"30735d0d4e57890bebb4c6250df7d4d635a462edc313f6f861fad392edce0bb1"}
Jul 07 01:01:36.398857 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"bd33f2e766487c1ef3ac1672f8ba214f63847c9144b2df59e1661b1dfef4d293"}
Jul 07 01:01:36.398857 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"8bc954ce5619515df24966068f2cb2f07b70cb3538a6513e94d67725f59d3ef8"}
Jul 07 01:01:36.398920 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"a9b2a3d33091e0b22d7e12c7868cf714d17d11fc392a432b4d426ad3b6efac63"}
Jul 07 01:01:36.398920 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerDied","Data":"0e9d558c61e5b002956e1337d1c1c3ad801dcdfc766a5644d8c4ff7678064673"}
Jul 07 01:01:36.398975 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.398930    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"cfc622862ca5fa85ef1b3129834dde552c720c5c506e41730019cb51ad42dee4"}
Jul 07 01:01:36.405108 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.405042    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-16.ec2.internal" podStartSLOduration=23.405030915 podStartE2EDuration="23.405030915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:36.404642275 +0000 UTC m=+24.679633597" watchObservedRunningTime="2026-07-07 01:01:36.405030915 +0000 UTC m=+24.680022237"
Jul 07 01:01:36.435530 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.434661    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dd47n" podStartSLOduration=3.203740593 podStartE2EDuration="24.434645694s" totalImagesPullingTime="21.230905101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:36.419916619 +0000 UTC m=+24.694907944" watchObservedRunningTime="2026-07-07 01:01:36.434645694 +0000 UTC m=+24.709637017"
Jul 07 01:01:36.435819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:36.435794    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v2ml8" podStartSLOduration=3.175213645 podStartE2EDuration="24.435786568s" totalImagesPullingTime="21.260572923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:36.435574815 +0000 UTC m=+24.710566119" watchObservedRunningTime="2026-07-07 01:01:36.435786568 +0000 UTC m=+24.710777891"
Jul 07 01:01:37.283710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.283490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:37.283891 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:37.283794    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:37.402330 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.402301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-97djw" event={"ID":"b2d23b4d-300b-4264-b083-6172651d2548","Type":"ContainerStarted","Data":"7599b7e77b3ab843547987553080e7d4fefe85fa0af53ca283dd03dfaf34234a"}
Jul 07 01:01:37.404050 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.404022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-75844" event={"ID":"aa786802-732e-45a6-9420-7cfe2571fd09","Type":"ContainerStarted","Data":"66623b0b14b3f27ac3ed6f38a21ff651b1d60b5ca961dbd0fb5f92ddaf587f7e"}
Jul 07 01:01:37.406344 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.406314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz" event={"ID":"c8f50afc-7a89-485b-a262-52c1aa1ca181","Type":"ContainerStarted","Data":"26068bbe28cbac95a7230704820cd9ce063341e121df2f29b4c9fde8050cd9b3"}
Jul 07 01:01:37.409847 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.409825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7mcs" event={"ID":"06aa83ec-04af-4f19-a0d3-b17a6cc90ba3","Type":"ContainerStarted","Data":"984f350ce55e2db53dd1e10fd3fd259c63c4bbea8c9c5a3ff183ab3094bd38af"}
Jul 07 01:01:37.411277 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.411222    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ddjh7" event={"ID":"46bdeae8-a38b-4939-adbf-91f694cad4ca","Type":"ContainerStarted","Data":"4a1559a80e89f2bc6081110209b215fb9bba0d5cdd3fe4556bbf24676bbde92e"}
Jul 07 01:01:37.413274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.413251    2562 generic.go:356] "Generic (PLEG): container finished" podID="9dbb58ca2bc7089e0dfff1cca9c75a0d" containerID="47942fd7caf351558ebca2e70a9265cecd6156cdcf69c3dc369a54d1dac5a146" exitCode=0
Jul 07 01:01:37.413368 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.413322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" event={"ID":"9dbb58ca2bc7089e0dfff1cca9c75a0d","Type":"ContainerDied","Data":"47942fd7caf351558ebca2e70a9265cecd6156cdcf69c3dc369a54d1dac5a146"}
Jul 07 01:01:37.415153 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.415129    2562 generic.go:356] "Generic (PLEG): container finished" podID="8216316d-d40e-4cf4-8d4c-13974c5e5f74" containerID="9bedf3a584ddf4349fbcd02a249451caa7dcb90d83964269c20c11d7c0727aef" exitCode=0
Jul 07 01:01:37.415245 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.415208    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerDied","Data":"9bedf3a584ddf4349fbcd02a249451caa7dcb90d83964269c20c11d7c0727aef"}
Jul 07 01:01:37.429240 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.429203    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b7mcs" podStartSLOduration=4.233668145 podStartE2EDuration="25.429191042s" totalImagesPullingTime="21.195522897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:37.429013233 +0000 UTC m=+25.704004555" watchObservedRunningTime="2026-07-07 01:01:37.429191042 +0000 UTC m=+25.704182365"
Jul 07 01:01:37.429740 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.429677    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-97djw" podStartSLOduration=4.217842603 podStartE2EDuration="25.429666417s" totalImagesPullingTime="21.211823814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:37.415635149 +0000 UTC m=+25.690626471" watchObservedRunningTime="2026-07-07 01:01:37.429666417 +0000 UTC m=+25.704657739"
Jul 07 01:01:37.459745 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.459714    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-75844" podStartSLOduration=4.243654074 podStartE2EDuration="25.45970293s" totalImagesPullingTime="21.216048856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:37.459533994 +0000 UTC m=+25.734525329" watchObservedRunningTime="2026-07-07 01:01:37.45970293 +0000 UTC m=+25.734694234"
Jul 07 01:01:37.490381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.490337    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ddjh7" podStartSLOduration=12.426917435 podStartE2EDuration="25.490325498s" totalImagesPullingTime="13.063408063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:37.490066257 +0000 UTC m=+25.765057580" watchObservedRunningTime="2026-07-07 01:01:37.490325498 +0000 UTC m=+25.765316819"
Jul 07 01:01:37.565933 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.565870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:37.644873 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.644852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:37.836875 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:37.836856    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 01:01:38.206007 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.205883    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T01:01:37.83687304Z","UUID":"4514f7e0-a4a9-4c53-b94c-9d058386bf15","Handler":null,"Name":"","Endpoint":""}
Jul 07 01:01:38.207542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.207526    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 01:01:38.207640 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.207550    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 01:01:38.283461 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.283438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:38.283598 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:38.283572    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:38.283657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.283628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:38.283766 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:38.283725    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:38.419207 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.419177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" event={"ID":"9dbb58ca2bc7089e0dfff1cca9c75a0d","Type":"ContainerStarted","Data":"fcaeebb9e81316971fac236b2737ec759e4fb9b78755ce87e0828e9a70178a33"}
Jul 07 01:01:38.421315 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.421288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz" event={"ID":"c8f50afc-7a89-485b-a262-52c1aa1ca181","Type":"ContainerStarted","Data":"dafe003504481ae597c594bc5623f9b2255497cb53375c9ff99193d4acc7f065"}
Jul 07 01:01:38.432673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.432628    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-16.ec2.internal" podStartSLOduration=25.432616153 podStartE2EDuration="25.432616153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:38.4321288 +0000 UTC m=+26.707120122" watchObservedRunningTime="2026-07-07 01:01:38.432616153 +0000 UTC m=+26.707607475"
Jul 07 01:01:38.812273 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.812166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:38.869824 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.869790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:38.922992 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.922961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:38.985026 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.984988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:38.985588 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:38.985558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:39.283819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.283781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:39.283998 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:39.283930    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:39.426920 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.426896    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:01:39.427335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.427300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"f3c9b73453944f8380906a9a302b700b6f6d68f2df062822adf9263d52f1a646"}
Jul 07 01:01:39.427641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.427610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:39.428083 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.428064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ddjh7"
Jul 07 01:01:39.483261 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.483222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:39.877172 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.877136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:39.981814 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:39.981787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:40.261162 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:40.260968    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:40.283761 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:40.283734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:40.283761 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:40.283740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:40.283944 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:40.283841    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:40.283944 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:40.283894    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:40.359561 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:40.359532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:41.283355 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.283319    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:41.284238 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:41.283440    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:41.436324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.436299    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:01:41.436816 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.436785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"f2535b5314b60660dbc249f0afa050aa32dde161a2fa824d55870e0b62950892"}
Jul 07 01:01:41.437288 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.437261    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:41.437382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.437304    2562 scope.go:117] "RemoveContainer" containerID="0e9d558c61e5b002956e1337d1c1c3ad801dcdfc766a5644d8c4ff7678064673"
Jul 07 01:01:41.437382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.437345    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:41.437507 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.437394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:41.454485 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.454456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:41.454946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:41.454925    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:01:42.284434 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.284213    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:42.284881 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.284267    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:42.284881 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:42.284530    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:42.284881 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:42.284579    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:42.689165 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.689130    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqst9"]
Jul 07 01:01:42.689375 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.689275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:42.689434 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:42.689383    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:42.689836 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.689808    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gx5ms"]
Jul 07 01:01:42.689988 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.689897    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:42.690047 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:42.689985    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:42.692058 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.692035    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gf5c2"]
Jul 07 01:01:42.692169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.692118    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:42.692232 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:42.692214    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:42.811324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:42.811258    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:43.443662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:43.443573    2562 generic.go:356] "Generic (PLEG): container finished" podID="8216316d-d40e-4cf4-8d4c-13974c5e5f74" containerID="20d3a294a6f9e77cbbe4eeafe776ada5937eb9fc88cfd5dd750937976a55c07b" exitCode=0
Jul 07 01:01:43.444211 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:43.443694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerDied","Data":"20d3a294a6f9e77cbbe4eeafe776ada5937eb9fc88cfd5dd750937976a55c07b"}
Jul 07 01:01:43.448428 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:43.448295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz" event={"ID":"c8f50afc-7a89-485b-a262-52c1aa1ca181","Type":"ContainerStarted","Data":"9a8225860edfbcb06e3d63ebda66cd7a164b5ff3d936a88d923dd9b8f2c4bd33"}
Jul 07 01:01:43.453197 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:43.453181    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:01:43.453775 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:43.453753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" event={"ID":"d61cb9e3-fa2c-44da-880f-3ba4a2892b47","Type":"ContainerStarted","Data":"e590b8b12e9b1a5d4841e0135dece110d52cd6c8c74e1f49b795e085f88beb05"}
Jul 07 01:01:43.482157 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:43.482104    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8vqbz" podStartSLOduration=4.110038035 podStartE2EDuration="31.482091272s" totalImagesPullingTime="27.372053237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:43.481617899 +0000 UTC m=+31.756609221" watchObservedRunningTime="2026-07-07 01:01:43.482091272 +0000 UTC m=+31.757082608"
Jul 07 01:01:43.504540 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:43.504493    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k" podStartSLOduration=10.200853357 podStartE2EDuration="31.50448581s" totalImagesPullingTime="21.303632453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:43.503082551 +0000 UTC m=+31.778073873" watchObservedRunningTime="2026-07-07 01:01:43.50448581 +0000 UTC m=+31.779477131"
Jul 07 01:01:44.283401 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:44.283368    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:44.283570 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:44.283371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:44.283570 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:44.283487    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:44.283667 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:44.283371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:44.283667 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:44.283578    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:44.283781 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:44.283664    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:44.877788 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:44.877754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:45.895477 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:45.895448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:45.895856 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:45.895507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:45.895856 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:45.895591    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:45.895856 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:45.895658    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret podName:85ba5407-90de-4401-a47c-51227f2e1958 nodeName:}" failed. No retries permitted until 2026-07-07 01:02:17.8956426 +0000 UTC m=+66.170633900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret") pod "global-pull-secret-syncer-lqst9" (UID: "85ba5407-90de-4401-a47c-51227f2e1958") : object "kube-system"/"original-pull-secret" not registered
Jul 07 01:01:45.895856 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:45.895595    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:45.895856 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:45.895711    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs podName:2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d nodeName:}" failed. No retries permitted until 2026-07-07 01:02:17.89570416 +0000 UTC m=+66.170695460 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs") pod "network-metrics-daemon-gf5c2" (UID: "2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 01:01:46.096867 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:46.096645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:46.096961 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:46.096806    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 01:01:46.096961 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:46.096951    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 01:01:46.097062 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:46.096962    2562 projected.go:196] Error preparing data for projected volume kube-api-access-gk8gr for pod openshift-network-diagnostics/network-check-target-gx5ms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:46.097062 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:46.097003    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr podName:b57b8b8c-50df-48ce-83bb-7e5d175c3d8d nodeName:}" failed. No retries permitted until 2026-07-07 01:02:18.096989073 +0000 UTC m=+66.371980377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gk8gr" (UniqueName: "kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr") pod "network-check-target-gx5ms" (UID: "b57b8b8c-50df-48ce-83bb-7e5d175c3d8d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 01:01:46.284010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:46.283977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:46.284126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:46.283982    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:46.284126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:46.284097    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:46.284126 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:46.284097    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:46.284283 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:46.284223    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:46.284375 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:46.284347    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:46.462384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:46.462313    2562 generic.go:356] "Generic (PLEG): container finished" podID="8216316d-d40e-4cf4-8d4c-13974c5e5f74" containerID="5f1bfe194b514dee6f189b3c34f021fbdbf189b609026110e94cb6515043715c" exitCode=0
Jul 07 01:01:46.462384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:46.462356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerDied","Data":"5f1bfe194b514dee6f189b3c34f021fbdbf189b609026110e94cb6515043715c"}
Jul 07 01:01:47.468076 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:47.468048    2562 generic.go:356] "Generic (PLEG): container finished" podID="8216316d-d40e-4cf4-8d4c-13974c5e5f74" containerID="fbf7dd6c0926d88f1a470b45d00da90b6efc1760a7a45eb51fbe14163fed0e9f" exitCode=0
Jul 07 01:01:47.468502 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:47.468090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerDied","Data":"fbf7dd6c0926d88f1a470b45d00da90b6efc1760a7a45eb51fbe14163fed0e9f"}
Jul 07 01:01:47.490051 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:47.490025    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:47.581475 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:47.581443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:47.660761 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:47.660734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:47.959608 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:47.959546    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:48.035609 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.035587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:48.097017 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.096973    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:48.206552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.206524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:48.283635 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.283611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:48.283635 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.283624    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:48.283847 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.283627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:48.283847 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:48.283727    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqst9" podUID="85ba5407-90de-4401-a47c-51227f2e1958"
Jul 07 01:01:48.283847 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:48.283799    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gx5ms" podUID="b57b8b8c-50df-48ce-83bb-7e5d175c3d8d"
Jul 07 01:01:48.283977 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:48.283886    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gf5c2" podUID="2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d"
Jul 07 01:01:48.307189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.307165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:48.539275 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.539211    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-75844_aa786802-732e-45a6-9420-7cfe2571fd09/dns-node-resolver/0.log"
Jul 07 01:01:48.656259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.656218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:48.752605 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:48.752524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:49.501903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.501871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:49.520304 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.520284    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-16.ec2.internal" event="NodeReady"
Jul 07 01:01:49.520408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.520380    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 01:01:49.540515 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.540461    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-97djw_b2d23b4d-300b-4264-b083-6172651d2548/node-ca/0.log"
Jul 07 01:01:49.553610 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.553586    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9b5db8889-4rtfj"]
Jul 07 01:01:49.557466 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.557443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.560084 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.560066    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 01:01:49.560263 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.560250    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 01:01:49.560342 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.560253    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 01:01:49.560395 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.560353    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v8v29\""
Jul 07 01:01:49.565035 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.564952    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8g4vr"]
Jul 07 01:01:49.566028 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.566010    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 01:01:49.568807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.568792    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hlgwz"]
Jul 07 01:01:49.568917 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.568903    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:49.571249 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.571231    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4ln7z\""
Jul 07 01:01:49.571352 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.571310    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 01:01:49.571482 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.571467    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 01:01:49.571615 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.571600    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 01:01:49.572821 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.572802    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9b5db8889-4rtfj"]
Jul 07 01:01:49.572917 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.572901    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.575624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.575395    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 01:01:49.575624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.575436    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 01:01:49.575624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.575394    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 01:01:49.575624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.575441    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 01:01:49.575624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.575396    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-67m22\""
Jul 07 01:01:49.577969 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.577950    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8g4vr"]
Jul 07 01:01:49.578962 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.578918    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hlgwz"]
Jul 07 01:01:49.580882 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.580864    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pnjxx"]
Jul 07 01:01:49.584709 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.584677    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.587394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.587378    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 01:01:49.587554 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.587537    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 01:01:49.587660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.587643    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9bjq\""
Jul 07 01:01:49.593197 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.593179    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnjxx"]
Jul 07 01:01:49.718959 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.718923    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:49.723988 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.723965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3d91970-b381-4618-b90b-3de5e92be502-crio-socket\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.724112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.723997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2z5b\" (UniqueName: \"kubernetes.io/projected/a0b885b9-6b31-4d8a-948e-bf60f787f84b-kube-api-access-q2z5b\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.724112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-image-registry-private-configuration\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.724112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724041    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-bound-sa-token\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.724112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.724286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b885b9-6b31-4d8a-948e-bf60f787f84b-config-volume\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.724286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60c72870-3b21-4e16-872e-2434bfdecd75-ca-trust-extracted\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.724286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.724286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3d91970-b381-4618-b90b-3de5e92be502-data-volume\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.724286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.724286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e1d0fd7-9197-4c07-91ce-852454ca238b-cert\") pod \"ingress-canary-8g4vr\" (UID: \"2e1d0fd7-9197-4c07-91ce-852454ca238b\") " pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:49.724473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgv2k\" (UniqueName: \"kubernetes.io/projected/b3d91970-b381-4618-b90b-3de5e92be502-kube-api-access-fgv2k\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.724473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.724473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0b885b9-6b31-4d8a-948e-bf60f787f84b-tmp-dir\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.724473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-installation-pull-secrets\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.724473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-trusted-ca\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.724473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpf8w\" (UniqueName: \"kubernetes.io/projected/2e1d0fd7-9197-4c07-91ce-852454ca238b-kube-api-access-hpf8w\") pod \"ingress-canary-8g4vr\" (UID: \"2e1d0fd7-9197-4c07-91ce-852454ca238b\") " pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:49.724473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvjn\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-kube-api-access-brvjn\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.724726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.724474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-registry-certificates\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.773057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.772996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:49.824987 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.824930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvjn\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-kube-api-access-brvjn\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825094 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.824982    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-registry-certificates\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825094 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3d91970-b381-4618-b90b-3de5e92be502-crio-socket\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.825094 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2z5b\" (UniqueName: \"kubernetes.io/projected/a0b885b9-6b31-4d8a-948e-bf60f787f84b-kube-api-access-q2z5b\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.825094 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-image-registry-private-configuration\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825094 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-bound-sa-token\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b885b9-6b31-4d8a-948e-bf60f787f84b-config-volume\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60c72870-3b21-4e16-872e-2434bfdecd75-ca-trust-extracted\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3d91970-b381-4618-b90b-3de5e92be502-data-volume\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e1d0fd7-9197-4c07-91ce-852454ca238b-cert\") pod \"ingress-canary-8g4vr\" (UID: \"2e1d0fd7-9197-4c07-91ce-852454ca238b\") " pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv2k\" (UniqueName: \"kubernetes.io/projected/b3d91970-b381-4618-b90b-3de5e92be502-kube-api-access-fgv2k\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825285    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3d91970-b381-4618-b90b-3de5e92be502-crio-socket\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.825343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0b885b9-6b31-4d8a-948e-bf60f787f84b-tmp-dir\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825381    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-installation-pull-secrets\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-trusted-ca\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825394    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b5db8889-4rtfj: secret "image-registry-tls" not found
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825452    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825493    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls podName:60c72870-3b21-4e16-872e-2434bfdecd75 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:50.325477787 +0000 UTC m=+38.600469087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls") pod "image-registry-9b5db8889-4rtfj" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75") : secret "image-registry-tls" not found
Jul 07 01:01:49.825802 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825509    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls podName:a0b885b9-6b31-4d8a-948e-bf60f787f84b nodeName:}" failed. No retries permitted until 2026-07-07 01:01:50.325500601 +0000 UTC m=+38.600491907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls") pod "dns-default-pnjxx" (UID: "a0b885b9-6b31-4d8a-948e-bf60f787f84b") : secret "dns-default-metrics-tls" not found
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825855    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3d91970-b381-4618-b90b-3de5e92be502-data-volume\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825864    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60c72870-3b21-4e16-872e-2434bfdecd75-ca-trust-extracted\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825859    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-registry-certificates\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.825896    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b885b9-6b31-4d8a-948e-bf60f787f84b-config-volume\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825918    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825979    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.825999    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:50.325980717 +0000 UTC m=+38.600972042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : configmap "kube-rbac-proxy" not found
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.826033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpf8w\" (UniqueName: \"kubernetes.io/projected/2e1d0fd7-9197-4c07-91ce-852454ca238b-kube-api-access-hpf8w\") pod \"ingress-canary-8g4vr\" (UID: \"2e1d0fd7-9197-4c07-91ce-852454ca238b\") " pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:49.826158 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:49.826041    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:50.326024548 +0000 UTC m=+38.601015863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:49.826563 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.826273    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0b885b9-6b31-4d8a-948e-bf60f787f84b-tmp-dir\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.826801 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.826782    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-trusted-ca\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.830198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.830176    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-installation-pull-secrets\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.830292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.830227    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-image-registry-private-configuration\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.830292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.830248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e1d0fd7-9197-4c07-91ce-852454ca238b-cert\") pod \"ingress-canary-8g4vr\" (UID: \"2e1d0fd7-9197-4c07-91ce-852454ca238b\") " pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:49.834597 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.834130    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2z5b\" (UniqueName: \"kubernetes.io/projected/a0b885b9-6b31-4d8a-948e-bf60f787f84b-kube-api-access-q2z5b\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:49.834597 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.834549    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgv2k\" (UniqueName: \"kubernetes.io/projected/b3d91970-b381-4618-b90b-3de5e92be502-kube-api-access-fgv2k\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:49.835043 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.835021    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpf8w\" (UniqueName: \"kubernetes.io/projected/2e1d0fd7-9197-4c07-91ce-852454ca238b-kube-api-access-hpf8w\") pod \"ingress-canary-8g4vr\" (UID: \"2e1d0fd7-9197-4c07-91ce-852454ca238b\") " pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:49.835205 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.835187    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-bound-sa-token\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.836889 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.836867    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvjn\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-kube-api-access-brvjn\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:49.881100 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:49.881080    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8g4vr"
Jul 07 01:01:50.028641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.028419    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8g4vr"]
Jul 07 01:01:50.034151 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:50.034125    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1d0fd7_9197_4c07_91ce_852454ca238b.slice/crio-4915fc8ec0b2ebaa9ecab1e66e54defa9f15145ec9b8fe27347b10cd2474c46f WatchSource:0}: Error finding container 4915fc8ec0b2ebaa9ecab1e66e54defa9f15145ec9b8fe27347b10cd2474c46f: Status 404 returned error can't find the container with id 4915fc8ec0b2ebaa9ecab1e66e54defa9f15145ec9b8fe27347b10cd2474c46f
Jul 07 01:01:50.256063 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.256031    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-r9zvn"]
Jul 07 01:01:50.282710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.282665    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-r9zvn"]
Jul 07 01:01:50.282829 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.282764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.283154 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.283136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:01:50.283347 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.283330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:01:50.283452 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.283331    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:01:50.285059 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.284985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:50.285932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.285914    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 01:01:50.286122 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286105    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 01:01:50.286200 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286123    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kwtz9\""
Jul 07 01:01:50.286259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286240    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 01:01:50.286334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286322    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 01:01:50.286450 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286438    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 01:01:50.286652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286613    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 01:01:50.286736 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286677    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 01:01:50.286798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286731    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6nlk5\""
Jul 07 01:01:50.286869 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.286852    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpzcw\""
Jul 07 01:01:50.287820 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.287101    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 01:01:50.330334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.330317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:50.330447 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.330364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:50.330447 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.330396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:50.330447 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.330441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330464    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330469    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330516    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330525    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:51.330506651 +0000 UTC m=+39.605497952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : configmap "kube-rbac-proxy" not found
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330534    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330543    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:51.330533533 +0000 UTC m=+39.605524834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330546    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b5db8889-4rtfj: secret "image-registry-tls" not found
Jul 07 01:01:50.330584 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330587    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls podName:60c72870-3b21-4e16-872e-2434bfdecd75 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:51.330572335 +0000 UTC m=+39.605563638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls") pod "image-registry-9b5db8889-4rtfj" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75") : secret "image-registry-tls" not found
Jul 07 01:01:50.330879 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:50.330627    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls podName:a0b885b9-6b31-4d8a-948e-bf60f787f84b nodeName:}" failed. No retries permitted until 2026-07-07 01:01:51.330620736 +0000 UTC m=+39.605612040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls") pod "dns-default-pnjxx" (UID: "a0b885b9-6b31-4d8a-948e-bf60f787f84b") : secret "dns-default-metrics-tls" not found
Jul 07 01:01:50.339115 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.339095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:50.431136 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.431115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcncg\" (UniqueName: \"kubernetes.io/projected/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-kube-api-access-mcncg\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.431238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.431168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-signing-key\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.431281 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.431249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-signing-cabundle\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.475660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.475628    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8g4vr" event={"ID":"2e1d0fd7-9197-4c07-91ce-852454ca238b","Type":"ContainerStarted","Data":"4915fc8ec0b2ebaa9ecab1e66e54defa9f15145ec9b8fe27347b10cd2474c46f"}
Jul 07 01:01:50.531923 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.531863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-signing-cabundle\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.532025 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.531932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcncg\" (UniqueName: \"kubernetes.io/projected/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-kube-api-access-mcncg\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.532025 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.531980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-signing-key\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.532600 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.532576    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-signing-cabundle\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.535818 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.535792    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-signing-key\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.539675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.539652    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcncg\" (UniqueName: \"kubernetes.io/projected/de7ad908-ab9e-40a8-808c-77e3eb1a3fdc-kube-api-access-mcncg\") pod \"service-ca-758f599c5-r9zvn\" (UID: \"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc\") " pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.597769 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.597744    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-r9zvn"
Jul 07 01:01:50.748264 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:50.748228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-r9zvn"]
Jul 07 01:01:50.751411 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:01:50.751386    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7ad908_ab9e_40a8_808c_77e3eb1a3fdc.slice/crio-59939f26683cb24cbc75e6f8742d3d734910ee07e5532e1d79c73cf549df95ac WatchSource:0}: Error finding container 59939f26683cb24cbc75e6f8742d3d734910ee07e5532e1d79c73cf549df95ac: Status 404 returned error can't find the container with id 59939f26683cb24cbc75e6f8742d3d734910ee07e5532e1d79c73cf549df95ac
Jul 07 01:01:51.132181 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:51.132142    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:51.338675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:51.338634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:51.338755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:51.338793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338838    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338875    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338916    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:53.338891902 +0000 UTC m=+41.613883203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338927    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338940    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b5db8889-4rtfj: secret "image-registry-tls" not found
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338941    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:53.33892425 +0000 UTC m=+41.613915563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : configmap "kube-rbac-proxy" not found
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:51.338842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338963    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 01:01:51.338995 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.338978    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls podName:60c72870-3b21-4e16-872e-2434bfdecd75 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:53.338966051 +0000 UTC m=+41.613957357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls") pod "image-registry-9b5db8889-4rtfj" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75") : secret "image-registry-tls" not found
Jul 07 01:01:51.339391 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:51.339014    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls podName:a0b885b9-6b31-4d8a-948e-bf60f787f84b nodeName:}" failed. No retries permitted until 2026-07-07 01:01:53.338999353 +0000 UTC m=+41.613990657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls") pod "dns-default-pnjxx" (UID: "a0b885b9-6b31-4d8a-948e-bf60f787f84b") : secret "dns-default-metrics-tls" not found
Jul 07 01:01:51.479201 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:51.479165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-r9zvn" event={"ID":"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc","Type":"ContainerStarted","Data":"59939f26683cb24cbc75e6f8742d3d734910ee07e5532e1d79c73cf549df95ac"}
Jul 07 01:01:52.000836 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:52.000804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:53.356803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.356518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:53.356803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.356801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:53.356700    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:53.356902    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls podName:a0b885b9-6b31-4d8a-948e-bf60f787f84b nodeName:}" failed. No retries permitted until 2026-07-07 01:01:57.356881354 +0000 UTC m=+45.631872663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls") pod "dns-default-pnjxx" (UID: "a0b885b9-6b31-4d8a-948e-bf60f787f84b") : secret "dns-default-metrics-tls" not found
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.356939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:53.356970    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.356977    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:53.356984    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b5db8889-4rtfj: secret "image-registry-tls" not found
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:53.357023    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:53.357032    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls podName:60c72870-3b21-4e16-872e-2434bfdecd75 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:57.357017154 +0000 UTC m=+45.632008468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls") pod "image-registry-9b5db8889-4rtfj" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75") : secret "image-registry-tls" not found
Jul 07 01:01:53.357282 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:53.357078    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:01:57.357063947 +0000 UTC m=+45.632055278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:53.357756 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.357485    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3d91970-b381-4618-b90b-3de5e92be502-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:53.488270 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.488237    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerStarted","Data":"b59b3059c412ef1a2d0a680beca2a18fdde1ba2b081f1240732851c9eb2355db"}
Jul 07 01:01:53.613431 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.613356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:53.859063 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.859035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:53.914735 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:53.914713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:54.493207 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:54.493164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8g4vr" event={"ID":"2e1d0fd7-9197-4c07-91ce-852454ca238b","Type":"ContainerStarted","Data":"3bc945484b9e2844f9c1fbed38ab96ab43ccd18dfd01b18a3c73d901f0db50e4"}
Jul 07 01:01:54.497057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:54.497024    2562 generic.go:356] "Generic (PLEG): container finished" podID="8216316d-d40e-4cf4-8d4c-13974c5e5f74" containerID="b59b3059c412ef1a2d0a680beca2a18fdde1ba2b081f1240732851c9eb2355db" exitCode=0
Jul 07 01:01:54.497200 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:54.497077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerDied","Data":"b59b3059c412ef1a2d0a680beca2a18fdde1ba2b081f1240732851c9eb2355db"}
Jul 07 01:01:54.507973 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:54.507934    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8g4vr" podStartSLOduration=1.633763686 podStartE2EDuration="5.507921729s" totalImagesPullingTime="3.874158043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:54.507368462 +0000 UTC m=+42.782359784" watchObservedRunningTime="2026-07-07 01:01:54.507921729 +0000 UTC m=+42.782913050"
Jul 07 01:01:54.764513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:54.764491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:54.825423 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:54.825395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:55.502324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:55.502286    2562 generic.go:356] "Generic (PLEG): container finished" podID="8216316d-d40e-4cf4-8d4c-13974c5e5f74" containerID="4aabd8fcff04d8c3e856a8198361385b955688c80c5f55970f9a4558a0f3a680" exitCode=0
Jul 07 01:01:55.502757 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:55.502324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerDied","Data":"4aabd8fcff04d8c3e856a8198361385b955688c80c5f55970f9a4558a0f3a680"}
Jul 07 01:01:55.504004 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:55.503969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-r9zvn" event={"ID":"de7ad908-ab9e-40a8-808c-77e3eb1a3fdc","Type":"ContainerStarted","Data":"c8c5b2884bf796400f52533e9e9d1a1717f90ffb9a45f46bfe8200dbafff6bc4"}
Jul 07 01:01:55.540260 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:55.540207    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-r9zvn" podStartSLOduration=1.381970796 podStartE2EDuration="5.540190346s" totalImagesPullingTime="4.15821955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:55.538971603 +0000 UTC m=+43.813962925" watchObservedRunningTime="2026-07-07 01:01:55.540190346 +0000 UTC m=+43.815181669"
Jul 07 01:01:56.458402 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:56.458337    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:56.512930 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:56.512889    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" event={"ID":"8216316d-d40e-4cf4-8d4c-13974c5e5f74","Type":"ContainerStarted","Data":"e3d5931bfda2de730bb530ef2c8bed95eada6ee6542a9c3380cebb9dd09c82a0"}
Jul 07 01:01:56.536837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:56.536766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2fkb2" podStartSLOduration=9.10310531 podStartE2EDuration="44.536743912s" totalImagesPullingTime="35.433638602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:01:56.536363811 +0000 UTC m=+44.811355134" watchObservedRunningTime="2026-07-07 01:01:56.536743912 +0000 UTC m=+44.811735237"
Jul 07 01:01:56.540206 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:56.540167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:56.614521 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:56.614489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:56.709308 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:56.709218    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:56.812267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:56.812233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:57.390751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:57.390713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:01:57.390751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:57.390758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:01:57.390965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:57.390788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:01:57.390965 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:57.390886    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:57.390965 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:57.390907    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 01:01:57.390965 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:57.390917    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9b5db8889-4rtfj: secret "image-registry-tls" not found
Jul 07 01:01:57.390965 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:57.390886    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 01:01:57.390965 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:57.390958    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls podName:b3d91970-b381-4618-b90b-3de5e92be502 nodeName:}" failed. No retries permitted until 2026-07-07 01:02:05.390942075 +0000 UTC m=+53.665933378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hlgwz" (UID: "b3d91970-b381-4618-b90b-3de5e92be502") : secret "insights-runtime-extractor-tls" not found
Jul 07 01:01:57.391146 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:57.390977    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls podName:a0b885b9-6b31-4d8a-948e-bf60f787f84b nodeName:}" failed. No retries permitted until 2026-07-07 01:02:05.390966467 +0000 UTC m=+53.665957768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls") pod "dns-default-pnjxx" (UID: "a0b885b9-6b31-4d8a-948e-bf60f787f84b") : secret "dns-default-metrics-tls" not found
Jul 07 01:01:57.391146 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:01:57.390988    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls podName:60c72870-3b21-4e16-872e-2434bfdecd75 nodeName:}" failed. No retries permitted until 2026-07-07 01:02:05.390982678 +0000 UTC m=+53.665973978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls") pod "image-registry-9b5db8889-4rtfj" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75") : secret "image-registry-tls" not found
Jul 07 01:01:57.753372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:57.753332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:57.952847 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:57.952802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:58.008938 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:58.008859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:58.354758 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:58.354651    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:58.552966 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:58.552930    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:58.753787 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:58.753754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:01:59.551997 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:01:59.551965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:01.297288 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:01.297251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:01.397399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:01.397373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:01.580180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:01.580103    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:02.034137 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:02.034106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:02.135011 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:02.134985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:02.236450 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:02.236421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:02.554463 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:02.554442    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:02.763053 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:02.763035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:04.731266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:04.731237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:04.779506 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:04.779479    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:04.951796 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:04.951774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:05.446487 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.446459    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:02:05.446712 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.446500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:02:05.446712 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.446557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:02:05.450169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.450139    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3d91970-b381-4618-b90b-3de5e92be502-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlgwz\" (UID: \"b3d91970-b381-4618-b90b-3de5e92be502\") " pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:02:05.450169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.450152    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0b885b9-6b31-4d8a-948e-bf60f787f84b-metrics-tls\") pod \"dns-default-pnjxx\" (UID: \"a0b885b9-6b31-4d8a-948e-bf60f787f84b\") " pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:02:05.450169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.450164    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"image-registry-9b5db8889-4rtfj\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") " pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:02:05.468069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.468049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:02:05.486189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.486170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hlgwz"
Jul 07 01:02:05.492788 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.492755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:02:05.552189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.552121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:05.637967 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.637931    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9b5db8889-4rtfj"]
Jul 07 01:02:05.641437 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:05.641410    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c72870_3b21_4e16_872e_2434bfdecd75.slice/crio-ee114ad8f2e345e531178998c7f3be676e2307faa3d9f69b262371ba48703dbe WatchSource:0}: Error finding container ee114ad8f2e345e531178998c7f3be676e2307faa3d9f69b262371ba48703dbe: Status 404 returned error can't find the container with id ee114ad8f2e345e531178998c7f3be676e2307faa3d9f69b262371ba48703dbe
Jul 07 01:02:05.654662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.654635    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hlgwz"]
Jul 07 01:02:05.657299 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:05.657272    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d91970_b381_4618_b90b_3de5e92be502.slice/crio-4ef746cabbe09a87df24a81eb0020d2ed52735ef7d7af076a1168ae6e6902748 WatchSource:0}: Error finding container 4ef746cabbe09a87df24a81eb0020d2ed52735ef7d7af076a1168ae6e6902748: Status 404 returned error can't find the container with id 4ef746cabbe09a87df24a81eb0020d2ed52735ef7d7af076a1168ae6e6902748
Jul 07 01:02:05.678039 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.678017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pnjxx"]
Jul 07 01:02:05.680787 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:05.680762    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b885b9_6b31_4d8a_948e_bf60f787f84b.slice/crio-1fd1d65071035490e1166c3f2e507fac068d73def540be391996e8ccee7e0ce1 WatchSource:0}: Error finding container 1fd1d65071035490e1166c3f2e507fac068d73def540be391996e8ccee7e0ce1: Status 404 returned error can't find the container with id 1fd1d65071035490e1166c3f2e507fac068d73def540be391996e8ccee7e0ce1
Jul 07 01:02:05.950078 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:05.950053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:06.369730 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.369621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:06.540644 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.540608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj" event={"ID":"60c72870-3b21-4e16-872e-2434bfdecd75","Type":"ContainerStarted","Data":"78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8"}
Jul 07 01:02:06.540841 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.540669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj" event={"ID":"60c72870-3b21-4e16-872e-2434bfdecd75","Type":"ContainerStarted","Data":"ee114ad8f2e345e531178998c7f3be676e2307faa3d9f69b262371ba48703dbe"}
Jul 07 01:02:06.540841 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.540746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:02:06.543287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.543259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnjxx" event={"ID":"a0b885b9-6b31-4d8a-948e-bf60f787f84b","Type":"ContainerStarted","Data":"1fd1d65071035490e1166c3f2e507fac068d73def540be391996e8ccee7e0ce1"}
Jul 07 01:02:06.546106 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.546082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlgwz" event={"ID":"b3d91970-b381-4618-b90b-3de5e92be502","Type":"ContainerStarted","Data":"4ef746cabbe09a87df24a81eb0020d2ed52735ef7d7af076a1168ae6e6902748"}
Jul 07 01:02:06.560390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.560341    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj" podStartSLOduration=62.560323641 podStartE2EDuration="1m2.560323641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:06.558630013 +0000 UTC m=+54.833621338" watchObservedRunningTime="2026-07-07 01:02:06.560323641 +0000 UTC m=+54.835314966"
Jul 07 01:02:06.574644 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.574571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:06.951328 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:06.951284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:07.155452 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:07.155422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:10.389721 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.389665    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"]
Jul 07 01:02:10.392789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.392762    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"]
Jul 07 01:02:10.392915 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.392905    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.395569 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.395551    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"]
Jul 07 01:02:10.395723 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.395705    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.397165 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.396919    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 01:02:10.397165 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.396984    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 01:02:10.397165 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.396924    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 01:02:10.398429 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.398411    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 01:02:10.398765 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.398748    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2lvwj\""
Jul 07 01:02:10.399090 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.398952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 01:02:10.399966 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.399587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.402868 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.402730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 01:02:10.403026 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.402879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 01:02:10.403401 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.403168    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 01:02:10.403639 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.403614    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 01:02:10.405250 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.405229    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"]
Jul 07 01:02:10.407148 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.407129    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"]
Jul 07 01:02:10.413569 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.413551    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"]
Jul 07 01:02:10.425568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.425544    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9b5db8889-4rtfj"]
Jul 07 01:02:10.486000 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.485971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcbt\" (UniqueName: \"kubernetes.io/projected/25df7608-5371-479e-a814-f4eacfb90ba5-kube-api-access-qvcbt\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.486000 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.485999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-ca\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.486177 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25df7608-5371-479e-a814-f4eacfb90ba5-tmp\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.486177 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486041    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjrw\" (UniqueName: \"kubernetes.io/projected/672fbab3-77d8-4315-9b70-4f11f52c6587-kube-api-access-whjrw\") pod \"managed-serviceaccount-addon-agent-55ddd455c6-5m5hm\" (UID: \"672fbab3-77d8-4315-9b70-4f11f52c6587\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.486177 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486090    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-hub\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.486177 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/25df7608-5371-479e-a814-f4eacfb90ba5-klusterlet-config\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.486177 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.486351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.486351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daaf099-2478-46bb-878d-08aa3cd44044-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.486351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/672fbab3-77d8-4315-9b70-4f11f52c6587-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55ddd455c6-5m5hm\" (UID: \"672fbab3-77d8-4315-9b70-4f11f52c6587\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.486351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.486255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98pv\" (UniqueName: \"kubernetes.io/projected/1daaf099-2478-46bb-878d-08aa3cd44044-kube-api-access-f98pv\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.507864 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.507833    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:10.559928 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.559900    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlgwz" event={"ID":"b3d91970-b381-4618-b90b-3de5e92be502","Type":"ContainerStarted","Data":"afcaf13721be7845116d4b88c638397d0b370f475a21bfeeb62a6a24e114e464"}
Jul 07 01:02:10.587309 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/25df7608-5371-479e-a814-f4eacfb90ba5-klusterlet-config\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.587409 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.587409 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.587409 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daaf099-2478-46bb-878d-08aa3cd44044-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.587409 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/672fbab3-77d8-4315-9b70-4f11f52c6587-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55ddd455c6-5m5hm\" (UID: \"672fbab3-77d8-4315-9b70-4f11f52c6587\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.587598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98pv\" (UniqueName: \"kubernetes.io/projected/1daaf099-2478-46bb-878d-08aa3cd44044-kube-api-access-f98pv\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.587598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcbt\" (UniqueName: \"kubernetes.io/projected/25df7608-5371-479e-a814-f4eacfb90ba5-kube-api-access-qvcbt\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.587598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-ca\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.587598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25df7608-5371-479e-a814-f4eacfb90ba5-tmp\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.587598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjrw\" (UniqueName: \"kubernetes.io/projected/672fbab3-77d8-4315-9b70-4f11f52c6587-kube-api-access-whjrw\") pod \"managed-serviceaccount-addon-agent-55ddd455c6-5m5hm\" (UID: \"672fbab3-77d8-4315-9b70-4f11f52c6587\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.587598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.587586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-hub\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.588229 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.588153    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1daaf099-2478-46bb-878d-08aa3cd44044-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.588707 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.588662    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25df7608-5371-479e-a814-f4eacfb90ba5-tmp\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.590120 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.590097    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.590553 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.590519    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.590652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.590598    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-ca\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.590652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.590603    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/25df7608-5371-479e-a814-f4eacfb90ba5-klusterlet-config\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.591036 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.590753    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1daaf099-2478-46bb-878d-08aa3cd44044-hub\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.591036 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.590883    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/672fbab3-77d8-4315-9b70-4f11f52c6587-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55ddd455c6-5m5hm\" (UID: \"672fbab3-77d8-4315-9b70-4f11f52c6587\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.612699 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.612653    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcbt\" (UniqueName: \"kubernetes.io/projected/25df7608-5371-479e-a814-f4eacfb90ba5-kube-api-access-qvcbt\") pod \"klusterlet-addon-workmgr-54ff6d55cc-d7j9c\" (UID: \"25df7608-5371-479e-a814-f4eacfb90ba5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.612798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.612773    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98pv\" (UniqueName: \"kubernetes.io/projected/1daaf099-2478-46bb-878d-08aa3cd44044-kube-api-access-f98pv\") pod \"cluster-proxy-proxy-agent-56dbbf67d7-rjrkg\" (UID: \"1daaf099-2478-46bb-878d-08aa3cd44044\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.615803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.613987    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjrw\" (UniqueName: \"kubernetes.io/projected/672fbab3-77d8-4315-9b70-4f11f52c6587-kube-api-access-whjrw\") pod \"managed-serviceaccount-addon-agent-55ddd455c6-5m5hm\" (UID: \"672fbab3-77d8-4315-9b70-4f11f52c6587\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.619068 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.619037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:10.707724 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.707700    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:10.723971 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.723950    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:10.725931 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.725913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"
Jul 07 01:02:10.733751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.733728    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"
Jul 07 01:02:10.825562 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.825537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:10.854379 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.854326    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"]
Jul 07 01:02:10.863148 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:10.863121    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25df7608_5371_479e_a814_f4eacfb90ba5.slice/crio-8890a7b0f26a7c519fdc320729fc23c01f02b40704ab03c00c972bdc675ee252 WatchSource:0}: Error finding container 8890a7b0f26a7c519fdc320729fc23c01f02b40704ab03c00c972bdc675ee252: Status 404 returned error can't find the container with id 8890a7b0f26a7c519fdc320729fc23c01f02b40704ab03c00c972bdc675ee252
Jul 07 01:02:10.879197 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.877322    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm"]
Jul 07 01:02:10.882317 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:10.882293    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod672fbab3_77d8_4315_9b70_4f11f52c6587.slice/crio-7c4a3855acad51a1881b6258e7b0d96bc7b6fc06cf8c75a098c6524610a54b6d WatchSource:0}: Error finding container 7c4a3855acad51a1881b6258e7b0d96bc7b6fc06cf8c75a098c6524610a54b6d: Status 404 returned error can't find the container with id 7c4a3855acad51a1881b6258e7b0d96bc7b6fc06cf8c75a098c6524610a54b6d
Jul 07 01:02:10.891921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.891900    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg"]
Jul 07 01:02:10.894525 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:10.894503    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1daaf099_2478_46bb_878d_08aa3cd44044.slice/crio-af82111008c88eb3f5fd2041ce710f6be512e6a3c41c0272e3094d367c094149 WatchSource:0}: Error finding container af82111008c88eb3f5fd2041ce710f6be512e6a3c41c0272e3094d367c094149: Status 404 returned error can't find the container with id af82111008c88eb3f5fd2041ce710f6be512e6a3c41c0272e3094d367c094149
Jul 07 01:02:10.908052 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:10.908007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:11.251630 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.251604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:11.352140 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.352110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:11.564282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.564212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg" event={"ID":"1daaf099-2478-46bb-878d-08aa3cd44044","Type":"ContainerStarted","Data":"af82111008c88eb3f5fd2041ce710f6be512e6a3c41c0272e3094d367c094149"}
Jul 07 01:02:11.565839 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.565814    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnjxx" event={"ID":"a0b885b9-6b31-4d8a-948e-bf60f787f84b","Type":"ContainerStarted","Data":"0918954704c7c802c32cfb772b2ca9ca20423e33ace319d3e8debac3b17da792"}
Jul 07 01:02:11.565953 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.565890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pnjxx" event={"ID":"a0b885b9-6b31-4d8a-948e-bf60f787f84b","Type":"ContainerStarted","Data":"98a46238891bc21d10c47592cb8426e9b95a091b90ef9e66313fc3005aee6bcc"}
Jul 07 01:02:11.565953 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.565926    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:02:11.567023 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.567000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm" event={"ID":"672fbab3-77d8-4315-9b70-4f11f52c6587","Type":"ContainerStarted","Data":"7c4a3855acad51a1881b6258e7b0d96bc7b6fc06cf8c75a098c6524610a54b6d"}
Jul 07 01:02:11.567940 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.567921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c" event={"ID":"25df7608-5371-479e-a814-f4eacfb90ba5","Type":"ContainerStarted","Data":"8890a7b0f26a7c519fdc320729fc23c01f02b40704ab03c00c972bdc675ee252"}
Jul 07 01:02:11.582647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:11.582608    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pnjxx" podStartSLOduration=17.755306676 podStartE2EDuration="22.582595594s" totalImagesPullingTime="4.827288918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:11.580877068 +0000 UTC m=+59.855868401" watchObservedRunningTime="2026-07-07 01:02:11.582595594 +0000 UTC m=+59.857586916"
Jul 07 01:02:12.125595 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:12.125563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:12.178924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:12.178870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:12.312195 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:12.312158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:12.367646 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:12.367574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:12.497603 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:12.497560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:12.576913 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:12.576592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlgwz" event={"ID":"b3d91970-b381-4618-b90b-3de5e92be502","Type":"ContainerStarted","Data":"01e4186cc66dbe2226312777354568781c0c8451ba8e5fabe032ab21efa87148"}
Jul 07 01:02:12.706392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:12.705297    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:13.250659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:13.250613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:14.477913 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:14.477880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t4h9k"
Jul 07 01:02:14.630134 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:14.630058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:14.732010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:14.731870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:15.327527 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:15.327493    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:15.431590 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:15.431551    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:15.496676 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:15.496568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:15.686907 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:15.686815    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:15.746270 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:15.746236    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:16.900074 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:16.900032    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:17.000529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:17.000488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:17.963536 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:17.963492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:02:17.963536 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:17.963536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:02:17.966247 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:17.966225    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 01:02:17.966373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:17.966262    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 01:02:17.976702 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:17.976661    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d-metrics-certs\") pod \"network-metrics-daemon-gf5c2\" (UID: \"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d\") " pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:02:17.976702 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:17.976675    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85ba5407-90de-4401-a47c-51227f2e1958-original-pull-secret\") pod \"global-pull-secret-syncer-lqst9\" (UID: \"85ba5407-90de-4401-a47c-51227f2e1958\") " pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:02:18.164807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.164772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:02:18.167360 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.167329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 01:02:18.178075 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.178050    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 01:02:18.189259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.189235    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8gr\" (UniqueName: \"kubernetes.io/projected/b57b8b8c-50df-48ce-83bb-7e5d175c3d8d-kube-api-access-gk8gr\") pod \"network-check-target-gx5ms\" (UID: \"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d\") " pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:02:18.203385 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.203349    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:18.205230 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.205209    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6nlk5\""
Jul 07 01:02:18.210358 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.210339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqst9"
Jul 07 01:02:18.213475 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.213460    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gf5c2"
Jul 07 01:02:18.217479 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.217461    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cpzcw\""
Jul 07 01:02:18.226313 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.226297    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:02:18.250958 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.250904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:18.649465 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.649379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gx5ms"]
Jul 07 01:02:18.657589 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:18.657520    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57b8b8c_50df_48ce_83bb_7e5d175c3d8d.slice/crio-8961804693bef9c64c8562e1396ade908b2bfea6b30cb0874deefd16ee3ef886 WatchSource:0}: Error finding container 8961804693bef9c64c8562e1396ade908b2bfea6b30cb0874deefd16ee3ef886: Status 404 returned error can't find the container with id 8961804693bef9c64c8562e1396ade908b2bfea6b30cb0874deefd16ee3ef886
Jul 07 01:02:18.678299 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.678274    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqst9"]
Jul 07 01:02:18.691344 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:18.691318    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gf5c2"]
Jul 07 01:02:18.696926 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:18.696879    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc84f2c_8889_4a9f_9bbe_da0aa52cab4d.slice/crio-013a3638af38e517f2cc03c9ccf37ab498954c01be0b9d28cc2110ec9e9977df WatchSource:0}: Error finding container 013a3638af38e517f2cc03c9ccf37ab498954c01be0b9d28cc2110ec9e9977df: Status 404 returned error can't find the container with id 013a3638af38e517f2cc03c9ccf37ab498954c01be0b9d28cc2110ec9e9977df
Jul 07 01:02:19.617787 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.617748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gf5c2" event={"ID":"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d","Type":"ContainerStarted","Data":"013a3638af38e517f2cc03c9ccf37ab498954c01be0b9d28cc2110ec9e9977df"}
Jul 07 01:02:19.620629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.620536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg" event={"ID":"1daaf099-2478-46bb-878d-08aa3cd44044","Type":"ContainerStarted","Data":"c32a6ba877735986511ffa2107e923117bf0d5a47f0ae3c29b32b02b1aeb05c5"}
Jul 07 01:02:19.620629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.620580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg" event={"ID":"1daaf099-2478-46bb-878d-08aa3cd44044","Type":"ContainerStarted","Data":"ac7908d74487bc987f8beec6b55e1bd1b84ea8d4e33375782edc1798cdb60c57"}
Jul 07 01:02:19.620629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.620600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg" event={"ID":"1daaf099-2478-46bb-878d-08aa3cd44044","Type":"ContainerStarted","Data":"a5fc1991c8d607e3085983bd1c47d12a8952052bd0176d5ece4b652005796696"}
Jul 07 01:02:19.623144 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.623114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlgwz" event={"ID":"b3d91970-b381-4618-b90b-3de5e92be502","Type":"ContainerStarted","Data":"64ce03f1a1593318b4c3783585f99c44a94ade63c573c2ba64e6103c6d10f8c3"}
Jul 07 01:02:19.625509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.625482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm" event={"ID":"672fbab3-77d8-4315-9b70-4f11f52c6587","Type":"ContainerStarted","Data":"de32e8d32340a75962857e0b97cd5e857721dc82145d9a0c6d9dac36aa53160b"}
Jul 07 01:02:19.627370 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.627313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gx5ms" event={"ID":"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d","Type":"ContainerStarted","Data":"8961804693bef9c64c8562e1396ade908b2bfea6b30cb0874deefd16ee3ef886"}
Jul 07 01:02:19.629057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.629032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c" event={"ID":"25df7608-5371-479e-a814-f4eacfb90ba5","Type":"ContainerStarted","Data":"40c594c8b625c8a78760b259ab66bd0ae146d5a58bc1a953b1133da0124cdd8d"}
Jul 07 01:02:19.629333 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.629309    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:19.631038 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.631009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c"
Jul 07 01:02:19.632069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.632026    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqst9" event={"ID":"85ba5407-90de-4401-a47c-51227f2e1958","Type":"ContainerStarted","Data":"63bb00349d1b5f6db2dd9ea4635d2698f2b6714326cbe7a3db4a111a38fdcb7e"}
Jul 07 01:02:19.638815 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.637919    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56dbbf67d7-rjrkg" podStartSLOduration=2.053513863 podStartE2EDuration="9.637902405s" totalImagesPullingTime="7.584388542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:19.636498285 +0000 UTC m=+67.911489599" watchObservedRunningTime="2026-07-07 01:02:19.637902405 +0000 UTC m=+67.912893726"
Jul 07 01:02:19.657005 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.656141    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hlgwz" podStartSLOduration=17.979545106 podStartE2EDuration="30.656126896s" totalImagesPullingTime="12.67658179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:19.655349146 +0000 UTC m=+67.930340479" watchObservedRunningTime="2026-07-07 01:02:19.656126896 +0000 UTC m=+67.931118218"
Jul 07 01:02:19.671512 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.670862    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55ddd455c6-5m5hm" podStartSLOduration=2.085253651 podStartE2EDuration="9.67084439s" totalImagesPullingTime="7.585590739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:19.669607933 +0000 UTC m=+67.944599256" watchObservedRunningTime="2026-07-07 01:02:19.67084439 +0000 UTC m=+67.945835741"
Jul 07 01:02:19.685394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:19.685348    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54ff6d55cc-d7j9c" podStartSLOduration=2.051256865 podStartE2EDuration="9.68533225s" totalImagesPullingTime="7.634075385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:19.684929873 +0000 UTC m=+67.959921196" watchObservedRunningTime="2026-07-07 01:02:19.68533225 +0000 UTC m=+67.960323576"
Jul 07 01:02:20.433297 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:20.433255    2562 patch_prober.go:28] interesting pod/image-registry-9b5db8889-4rtfj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 01:02:20.433487 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:20.433320    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj" podUID="60c72870-3b21-4e16-872e-2434bfdecd75" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 01:02:20.638268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:20.638170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gf5c2" event={"ID":"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d","Type":"ContainerStarted","Data":"58ac4549b8def145048afbbe3adeba4efe1aab6cc5c9a41bbbdc19a191a11c10"}
Jul 07 01:02:20.638268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:20.638237    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gf5c2" event={"ID":"2dc84f2c-8889-4a9f-9bbe-da0aa52cab4d","Type":"ContainerStarted","Data":"dcb4893fe61458611e9582f21d6c1eb278d9e55f8ed58a47d778d06037c5e8ac"}
Jul 07 01:02:20.653941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:20.653891    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gf5c2" podStartSLOduration=67.543924697 podStartE2EDuration="1m8.653873446s" totalImagesPullingTime="1.109948749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:20.653113099 +0000 UTC m=+68.928104425" watchObservedRunningTime="2026-07-07 01:02:20.653873446 +0000 UTC m=+68.928864769"
Jul 07 01:02:20.742696 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:20.742652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:20.792101 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:20.792059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:21.579025 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:21.578993    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pnjxx"
Jul 07 01:02:22.647966 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:22.647867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gx5ms" event={"ID":"b57b8b8c-50df-48ce-83bb-7e5d175c3d8d","Type":"ContainerStarted","Data":"d1463560cb47f88576d23d609b4833ba7331a436c90fa67f3d465a81b3e1be4d"}
Jul 07 01:02:22.648333 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:22.648004    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:02:22.662658 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:22.662601    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gx5ms" podStartSLOduration=66.970945445 podStartE2EDuration="1m10.662586315s" totalImagesPullingTime="3.69164087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:22.661784207 +0000 UTC m=+70.936775529" watchObservedRunningTime="2026-07-07 01:02:22.662586315 +0000 UTC m=+70.937577638"
Jul 07 01:02:23.520180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:23.520138    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:23.580550 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:23.580525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:23.835625 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:23.835534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:25.660642 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:25.660611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqst9" event={"ID":"85ba5407-90de-4401-a47c-51227f2e1958","Type":"ContainerStarted","Data":"24bbf02101f53d67eee5bcdfc6c6383bfe2ef18c00175ef4fdbd4ce01cee323c"}
Jul 07 01:02:25.679252 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:25.679203    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lqst9" podStartSLOduration=67.603566865 podStartE2EDuration="1m13.679185505s" totalImagesPullingTime="6.07561864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:25.678532968 +0000 UTC m=+73.953524289" watchObservedRunningTime="2026-07-07 01:02:25.679185505 +0000 UTC m=+73.954176828"
Jul 07 01:02:26.123450 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:26.123420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:26.178436 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:26.178413    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:30.431079 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:30.431049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:02:30.690141 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:30.690032    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:30.790373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:30.790341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:30.890603 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:30.890562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:33.201588 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:33.201542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:33.251740 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:33.251678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:33.994741 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:33.994616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:34.052402 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:34.052381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:35.451606 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.451565    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj" podUID="60c72870-3b21-4e16-872e-2434bfdecd75" containerName="registry" containerID="cri-o://78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8" gracePeriod=30
Jul 07 01:02:35.688207 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.688184    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:02:35.693867 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.693841    2562 generic.go:356] "Generic (PLEG): container finished" podID="60c72870-3b21-4e16-872e-2434bfdecd75" containerID="78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8" exitCode=0
Jul 07 01:02:35.693961 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.693880    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj"
Jul 07 01:02:35.694003 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.693882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj" event={"ID":"60c72870-3b21-4e16-872e-2434bfdecd75","Type":"ContainerDied","Data":"78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8"}
Jul 07 01:02:35.694037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.694015    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9b5db8889-4rtfj" event={"ID":"60c72870-3b21-4e16-872e-2434bfdecd75","Type":"ContainerDied","Data":"ee114ad8f2e345e531178998c7f3be676e2307faa3d9f69b262371ba48703dbe"}
Jul 07 01:02:35.694068 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.694039    2562 scope.go:117] "RemoveContainer" containerID="78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8"
Jul 07 01:02:35.701428 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.701409    2562 scope.go:117] "RemoveContainer" containerID="78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8"
Jul 07 01:02:35.701753 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:02:35.701708    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8\": container with ID starting with 78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8 not found: ID does not exist" containerID="78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8"
Jul 07 01:02:35.701820 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.701748    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8"} err="failed to get container status \"78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8\": rpc error: code = NotFound desc = could not find container \"78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8\": container with ID starting with 78694ee91bf9c42133649f1bb2b8d8a0e4bc39bbb5197850c1b4ea1190b977a8 not found: ID does not exist"
Jul 07 01:02:35.790093 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790067    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-trusted-ca\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790203 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790099    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brvjn\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-kube-api-access-brvjn\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790203 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790120    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790203 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790142    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-image-registry-private-configuration\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790203 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-registry-certificates\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790208    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-installation-pull-secrets\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-bound-sa-token\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790289    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60c72870-3b21-4e16-872e-2434bfdecd75-ca-trust-extracted\") pod \"60c72870-3b21-4e16-872e-2434bfdecd75\" (UID: \"60c72870-3b21-4e16-872e-2434bfdecd75\") "
Jul 07 01:02:35.790593 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.790544    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:02:35.791190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.791150    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:02:35.792877 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.792829    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:02:35.793048 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.793023    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:02:35.793293 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.793269    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-kube-api-access-brvjn" (OuterVolumeSpecName: "kube-api-access-brvjn") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "kube-api-access-brvjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:02:35.793353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.793298    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:02:35.793391 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.793345    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:02:35.798509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.798478    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c72870-3b21-4e16-872e-2434bfdecd75-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "60c72870-3b21-4e16-872e-2434bfdecd75" (UID: "60c72870-3b21-4e16-872e-2434bfdecd75"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:02:35.891000 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.890975    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-installation-pull-secrets\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:35.891099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.890999    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-bound-sa-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:35.891099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.891016    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60c72870-3b21-4e16-872e-2434bfdecd75-ca-trust-extracted\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:35.891099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.891028    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-trusted-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:35.891099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.891040    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvjn\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-kube-api-access-brvjn\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:35.891099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.891052    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60c72870-3b21-4e16-872e-2434bfdecd75-registry-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:35.891099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.891065    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60c72870-3b21-4e16-872e-2434bfdecd75-image-registry-private-configuration\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:35.891099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:35.891078    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60c72870-3b21-4e16-872e-2434bfdecd75-registry-certificates\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:02:36.015836 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:36.015812    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9b5db8889-4rtfj"]
Jul 07 01:02:36.021083 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:36.021059    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9b5db8889-4rtfj"]
Jul 07 01:02:36.148649 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:36.148623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:36.201054 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:36.201033    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:36.288196 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:36.288146    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c72870-3b21-4e16-872e-2434bfdecd75" path="/var/lib/kubelet/pods/60c72870-3b21-4e16-872e-2434bfdecd75/volumes"
Jul 07 01:02:37.724117 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:37.724084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:37.834237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:37.834209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:37.934552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:37.934520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:38.450717 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:38.450663    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:38.551789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:38.551763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:39.638657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.638628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:39.738675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.738644    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:39.848911 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.848884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:39.882384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.882359    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mpxgb"]
Jul 07 01:02:39.882596 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.882584    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c72870-3b21-4e16-872e-2434bfdecd75" containerName="registry"
Jul 07 01:02:39.882643 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.882598    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c72870-3b21-4e16-872e-2434bfdecd75" containerName="registry"
Jul 07 01:02:39.882701 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.882645    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c72870-3b21-4e16-872e-2434bfdecd75" containerName="registry"
Jul 07 01:02:39.887350 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.887334    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:39.889675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.889614    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 01:02:39.889817 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.889652    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 01:02:39.889963 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.889744    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v6mvv\""
Jul 07 01:02:39.890107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.890093    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 01:02:39.890952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.890935    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 01:02:39.891049 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.890975    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 01:02:39.891110 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.890935    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 01:02:39.936784 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:39.936760    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.002311 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.002293    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.019453 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-root\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-wtmp\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-tls\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019592    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-textfile\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-metrics-client-ca\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019827 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbv4\" (UniqueName: \"kubernetes.io/projected/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-kube-api-access-kzbv4\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.019827 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.019791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-sys\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120565 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-sys\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120702 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120702 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-root\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120702 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-wtmp\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120702 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120663    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-tls\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120735    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-sys\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-textfile\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-metrics-client-ca\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbv4\" (UniqueName: \"kubernetes.io/projected/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-kube-api-access-kzbv4\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120821    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-wtmp\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.120924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.120883    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-root\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.121264 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:02:40.120961    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 01:02:40.121264 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:02:40.121013    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-tls podName:a3da0820-4c8d-4380-b8fe-0c360f2d45c3 nodeName:}" failed. No retries permitted until 2026-07-07 01:02:40.620994973 +0000 UTC m=+88.895986288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-tls") pod "node-exporter-mpxgb" (UID: "a3da0820-4c8d-4380-b8fe-0c360f2d45c3") : secret "node-exporter-tls" not found
Jul 07 01:02:40.121264 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.121040    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-textfile\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.121498 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.121467    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-metrics-client-ca\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.121695 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.121660    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.123548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.123527    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.131344 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.131321    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbv4\" (UniqueName: \"kubernetes.io/projected/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-kube-api-access-kzbv4\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.132111 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.132076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.225988 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.225970    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.327798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.327767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.418264 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.418223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.475812 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.475790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.573928 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.573881    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.623664 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.623644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-tls\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.625803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.625776    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3da0820-4c8d-4380-b8fe-0c360f2d45c3-node-exporter-tls\") pod \"node-exporter-mpxgb\" (UID: \"a3da0820-4c8d-4380-b8fe-0c360f2d45c3\") " pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.798789 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.798767    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mpxgb"
Jul 07 01:02:40.833330 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:02:40.833307    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3da0820_4c8d_4380_b8fe_0c360f2d45c3.slice/crio-728f9ca1dbaa6af709b9979213a311835b40eb99f679b421fc95cab52196075c WatchSource:0}: Error finding container 728f9ca1dbaa6af709b9979213a311835b40eb99f679b421fc95cab52196075c: Status 404 returned error can't find the container with id 728f9ca1dbaa6af709b9979213a311835b40eb99f679b421fc95cab52196075c
Jul 07 01:02:40.856738 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.856715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:40.943774 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:40.943750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:41.087269 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:41.087179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:41.193575 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:41.193553    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:41.251663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:41.251629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:41.715898 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:41.715857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpxgb" event={"ID":"a3da0820-4c8d-4380-b8fe-0c360f2d45c3","Type":"ContainerStarted","Data":"728f9ca1dbaa6af709b9979213a311835b40eb99f679b421fc95cab52196075c"}
Jul 07 01:02:41.750198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:41.750160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:42.720494 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:42.720465    2562 generic.go:356] "Generic (PLEG): container finished" podID="a3da0820-4c8d-4380-b8fe-0c360f2d45c3" containerID="de225a7cf4f6dbb00caba5afd6a8920f146489acd6d5bcbc1218a49dc9edeacb" exitCode=0
Jul 07 01:02:42.720808 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:42.720556    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpxgb" event={"ID":"a3da0820-4c8d-4380-b8fe-0c360f2d45c3","Type":"ContainerDied","Data":"de225a7cf4f6dbb00caba5afd6a8920f146489acd6d5bcbc1218a49dc9edeacb"}
Jul 07 01:02:42.846235 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:42.846206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:42.947565 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:42.947542    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:43.047074 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:43.046993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:43.147412 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:43.147384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:43.726471 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:43.726413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpxgb" event={"ID":"a3da0820-4c8d-4380-b8fe-0c360f2d45c3","Type":"ContainerStarted","Data":"2efc393efaf04e62fd2312774086dec93df0fdda74198a0ba79162389599924a"}
Jul 07 01:02:43.726972 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:43.726486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mpxgb" event={"ID":"a3da0820-4c8d-4380-b8fe-0c360f2d45c3","Type":"ContainerStarted","Data":"09d34da965e5985c993211b11c5869f2b58eb731fb844fbb03203be85d0bd689"}
Jul 07 01:02:43.747649 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:43.747594    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mpxgb" podStartSLOduration=3.249549001 podStartE2EDuration="4.747581231s" totalImagesPullingTime="1.388572954s" totalInitContainerRuntime="109.459276ms" isStatefulPod=true podCreationTimestamp="2026-07-07 01:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:02:43.745898443 +0000 UTC m=+92.020889757" watchObservedRunningTime="2026-07-07 01:02:43.747581231 +0000 UTC m=+92.022572552"
Jul 07 01:02:44.695049 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:44.695015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:44.766542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:44.766501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:45.484255 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:45.484211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:45.587524 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:45.587492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:46.133037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:46.132995    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:47.530429 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:47.530392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:47.582377 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:47.582341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:48.203938 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:48.203907    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:48.258445 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:48.258403    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:02:53.663408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:02:53.663375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gx5ms"
Jul 07 01:03:03.209124 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:03.209086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:03.308493 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:03.308460    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:04.882193 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:04.882154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:04.932135 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:04.932100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:04.982190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:04.982162    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:10.618001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:10.617960    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:10.717842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:10.717810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:10.773911 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:10.773884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:55.947941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:55.947905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:03:55.997105 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:03:55.997070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:07.972143 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:07.972052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:08.022141 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:08.022109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:10.709048 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:10.709015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:10.760666 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:10.760632    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:10.821840 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:10.821810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:10.922928 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:10.922894    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:11.630929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:11.630899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:11.732524 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:11.732492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:18.222529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:18.222449    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:18.321447 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:18.321411    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:21.201286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:21.201246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:46.166929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:46.166891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:46.236206 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:46.236172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:04:56.534748 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:04:56.534705    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:00.716325 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:00.716290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:12.666460 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:12.666418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:13.423172 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:13.423134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:16.248516 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:16.248480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:16.298170 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:16.298139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:17.683536 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:17.683503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:17.779598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:17.779569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:18.556138 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:18.556109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:18.653185 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:18.653153    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:20.565954 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:20.565922    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:20.962648 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:20.962567    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:23.043402 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:23.043367    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:23.099957 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:23.099929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:25.287110 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:25.287065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:25.390932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:25.390899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:29.158749 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:29.158709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:30.525054 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:30.525014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:30.622941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:30.622880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:32.245175 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.245139    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"]
Jul 07 01:05:32.248357 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.248337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.253540 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.251036    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 01:05:32.253540 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.252229    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 01:05:32.253540 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.252500    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-m8fcd\""
Jul 07 01:05:32.256751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.256730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"]
Jul 07 01:05:32.358182 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.358139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26z4\" (UniqueName: \"kubernetes.io/projected/e447403b-0948-4743-b33c-d493bc7af52c-kube-api-access-s26z4\") pod \"openshift-lws-operator-fd8ccff4c-6bt2f\" (UID: \"e447403b-0948-4743-b33c-d493bc7af52c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.358355 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.358206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e447403b-0948-4743-b33c-d493bc7af52c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6bt2f\" (UID: \"e447403b-0948-4743-b33c-d493bc7af52c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.459155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.459122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e447403b-0948-4743-b33c-d493bc7af52c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6bt2f\" (UID: \"e447403b-0948-4743-b33c-d493bc7af52c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.459307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.459172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26z4\" (UniqueName: \"kubernetes.io/projected/e447403b-0948-4743-b33c-d493bc7af52c-kube-api-access-s26z4\") pod \"openshift-lws-operator-fd8ccff4c-6bt2f\" (UID: \"e447403b-0948-4743-b33c-d493bc7af52c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.459601 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.459580    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e447403b-0948-4743-b33c-d493bc7af52c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6bt2f\" (UID: \"e447403b-0948-4743-b33c-d493bc7af52c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.470798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.470775    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26z4\" (UniqueName: \"kubernetes.io/projected/e447403b-0948-4743-b33c-d493bc7af52c-kube-api-access-s26z4\") pod \"openshift-lws-operator-fd8ccff4c-6bt2f\" (UID: \"e447403b-0948-4743-b33c-d493bc7af52c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.564032 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.563957    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"
Jul 07 01:05:32.689417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:32.689339    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f"]
Jul 07 01:05:32.692189 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:05:32.692156    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode447403b_0948_4743_b33c_d493bc7af52c.slice/crio-cb1a287c6630897a9f695f32792dbc5702cdceb562aa39dbc49dfbf9540f667d WatchSource:0}: Error finding container cb1a287c6630897a9f695f32792dbc5702cdceb562aa39dbc49dfbf9540f667d: Status 404 returned error can't find the container with id cb1a287c6630897a9f695f32792dbc5702cdceb562aa39dbc49dfbf9540f667d
Jul 07 01:05:33.291606 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:33.291567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f" event={"ID":"e447403b-0948-4743-b33c-d493bc7af52c","Type":"ContainerStarted","Data":"cb1a287c6630897a9f695f32792dbc5702cdceb562aa39dbc49dfbf9540f667d"}
Jul 07 01:05:36.304014 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:36.303984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f" event={"ID":"e447403b-0948-4743-b33c-d493bc7af52c","Type":"ContainerStarted","Data":"ec071aac0eb257348bb7f0aefe2578b2de06ed04065e1f523da4403498d26d04"}
Jul 07 01:05:36.319493 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:36.319437    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6bt2f" podStartSLOduration=0.935576108 podStartE2EDuration="4.319418804s" totalImagesPullingTime="3.383842696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:05:36.317785242 +0000 UTC m=+264.592776562" watchObservedRunningTime="2026-07-07 01:05:36.319418804 +0000 UTC m=+264.594410126"
Jul 07 01:05:37.012359 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:37.012311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:37.112290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:37.112257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:39.500214 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:39.500182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:51.944776 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:51.944731    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:52.032701 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.032648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:52.489450 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.489370    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:52.723198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.723170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:52.828374 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.828283    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:52.854591 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.854562    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"]
Jul 07 01:05:52.861751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.861735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:52.864381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.864360    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 01:05:52.865042 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.865009    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 01:05:52.865042 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.865029    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 01:05:52.865210 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.865110    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qqjwn\""
Jul 07 01:05:52.865703 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.865666    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 01:05:52.878024 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.878005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"]
Jul 07 01:05:52.899277 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.899256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f738f51-3152-4dcf-b713-1e147ea2bcb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:52.899380 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.899294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpkk\" (UniqueName: \"kubernetes.io/projected/4f738f51-3152-4dcf-b713-1e147ea2bcb5-kube-api-access-nxpkk\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:52.899380 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.899316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f738f51-3152-4dcf-b713-1e147ea2bcb5-manifests\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:52.899380 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.899340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f738f51-3152-4dcf-b713-1e147ea2bcb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:52.899497 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.899393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f738f51-3152-4dcf-b713-1e147ea2bcb5-tmp\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:52.987526 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:52.987490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.000184 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.000158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f738f51-3152-4dcf-b713-1e147ea2bcb5-manifests\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.000276 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.000190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f738f51-3152-4dcf-b713-1e147ea2bcb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.000276 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.000211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f738f51-3152-4dcf-b713-1e147ea2bcb5-tmp\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.000276 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.000252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f738f51-3152-4dcf-b713-1e147ea2bcb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.000426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.000290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpkk\" (UniqueName: \"kubernetes.io/projected/4f738f51-3152-4dcf-b713-1e147ea2bcb5-kube-api-access-nxpkk\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.000549 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.000529    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f738f51-3152-4dcf-b713-1e147ea2bcb5-manifests\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.002579 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.002550    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f738f51-3152-4dcf-b713-1e147ea2bcb5-tmp\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.002891 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.002870    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f738f51-3152-4dcf-b713-1e147ea2bcb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.002979 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.002920    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f738f51-3152-4dcf-b713-1e147ea2bcb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.011491 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.011470    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpkk\" (UniqueName: \"kubernetes.io/projected/4f738f51-3152-4dcf-b713-1e147ea2bcb5-kube-api-access-nxpkk\") pod \"opendatahub-operator-controller-manager-7f9ffff4fc-45vqz\" (UID: \"4f738f51-3152-4dcf-b713-1e147ea2bcb5\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.084843 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.084762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.176199 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.176171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:53.184782 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.184753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.296626 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.296596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.311656 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.311630    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"]
Jul 07 01:05:53.314917 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:05:53.314896    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f738f51_3152_4dcf_b713_1e147ea2bcb5.slice/crio-69dd134801692ecbad596179914b3075afca1e1a8bb5867a86b533907e96f93a WatchSource:0}: Error finding container 69dd134801692ecbad596179914b3075afca1e1a8bb5867a86b533907e96f93a: Status 404 returned error can't find the container with id 69dd134801692ecbad596179914b3075afca1e1a8bb5867a86b533907e96f93a
Jul 07 01:05:53.362793 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.362738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz" event={"ID":"4f738f51-3152-4dcf-b713-1e147ea2bcb5","Type":"ContainerStarted","Data":"69dd134801692ecbad596179914b3075afca1e1a8bb5867a86b533907e96f93a"}
Jul 07 01:05:53.390165 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.390134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.488929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.488886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.585654 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.585627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.845279 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.845245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:53.949353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:53.949318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:05:57.379778 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:57.379743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz" event={"ID":"4f738f51-3152-4dcf-b713-1e147ea2bcb5","Type":"ContainerStarted","Data":"76213cb011eda23ad8847cea2d7e0b9d40db630ae68bc59a80c02cd459280342"}
Jul 07 01:05:58.386935 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:58.386899    2562 generic.go:356] "Generic (PLEG): container finished" podID="4f738f51-3152-4dcf-b713-1e147ea2bcb5" containerID="76213cb011eda23ad8847cea2d7e0b9d40db630ae68bc59a80c02cd459280342" exitCode=0
Jul 07 01:05:58.387324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:58.386971    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz" event={"ID":"4f738f51-3152-4dcf-b713-1e147ea2bcb5","Type":"ContainerDied","Data":"76213cb011eda23ad8847cea2d7e0b9d40db630ae68bc59a80c02cd459280342"}
Jul 07 01:05:59.392968 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:59.392889    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz" event={"ID":"4f738f51-3152-4dcf-b713-1e147ea2bcb5","Type":"ContainerStarted","Data":"ad78bc83e1fc96f43faa1b2c20ce43f48e0c667eead6ae6b31198d0df8603895"}
Jul 07 01:05:59.393309 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:59.393022    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:05:59.419193 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:05:59.419140    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz" podStartSLOduration=2.653874707 podStartE2EDuration="7.419125135s" totalImagesPullingTime="3.694491781s" totalInitContainerRuntime="1.070758647s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:05:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:05:59.417371712 +0000 UTC m=+287.692363055" watchObservedRunningTime="2026-07-07 01:05:59.419125135 +0000 UTC m=+287.694116456"
Jul 07 01:06:02.480160 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:02.480123    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:02.539517 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:02.539474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:04.886848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:04.886809    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:06.235143 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:06.235092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:07.763758 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:07.763715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:07.877341 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:07.877304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:10.399819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:10.399786    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f9ffff4fc-45vqz"
Jul 07 01:06:12.142323 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.142294    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:06:12.146920 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.146897    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 01:06:12.154857 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.154833    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:12.155476 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.155460    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:06:12.279170 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.279135    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:12.747390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.747360    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"]
Jul 07 01:06:12.750470 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.750453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.753323 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.753304    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-b5nqp\""
Jul 07 01:06:12.753323 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.753315    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 01:06:12.754417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.754404    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 01:06:12.754483 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.754409    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 01:06:12.756919 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.756902    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 01:06:12.768271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.768247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"]
Jul 07 01:06:12.835407 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.835387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da35937-89d5-48eb-bcb2-ef3d38e15c02-tmp\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.835522 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.835457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da35937-89d5-48eb-bcb2-ef3d38e15c02-tls-certs\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.835522 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.835505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvk5\" (UniqueName: \"kubernetes.io/projected/8da35937-89d5-48eb-bcb2-ef3d38e15c02-kube-api-access-8qvk5\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.841341 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.841322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:12.936557 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.936535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da35937-89d5-48eb-bcb2-ef3d38e15c02-tls-certs\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.936645 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.936562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvk5\" (UniqueName: \"kubernetes.io/projected/8da35937-89d5-48eb-bcb2-ef3d38e15c02-kube-api-access-8qvk5\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.936645 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.936610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da35937-89d5-48eb-bcb2-ef3d38e15c02-tmp\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.938852 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.938824    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da35937-89d5-48eb-bcb2-ef3d38e15c02-tmp\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.939030 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.939015    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da35937-89d5-48eb-bcb2-ef3d38e15c02-tls-certs\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.946100 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.946078    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6ntnl"]
Jul 07 01:06:12.949368 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.949354    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:12.951083 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.951064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:12.951838 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.951817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 07 01:06:12.951927 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.951844    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b6zz6\""
Jul 07 01:06:12.952417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.952396    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvk5\" (UniqueName: \"kubernetes.io/projected/8da35937-89d5-48eb-bcb2-ef3d38e15c02-kube-api-access-8qvk5\") pod \"kube-auth-proxy-9b59ff655-j9mqx\" (UID: \"8da35937-89d5-48eb-bcb2-ef3d38e15c02\") " pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:12.963868 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:12.963850    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6ntnl"]
Jul 07 01:06:13.019207 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.019153    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:13.037553 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.037526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kjq\" (UniqueName: \"kubernetes.io/projected/3b4ff3e6-74d0-443a-b28a-3a83b971fcd3-kube-api-access-b6kjq\") pod \"odh-model-controller-7998c67dc4-6ntnl\" (UID: \"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:13.037661 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.037645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b4ff3e6-74d0-443a-b28a-3a83b971fcd3-cert\") pod \"odh-model-controller-7998c67dc4-6ntnl\" (UID: \"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:13.059673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.059640    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"
Jul 07 01:06:13.125153 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.125111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:13.138840 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.138814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kjq\" (UniqueName: \"kubernetes.io/projected/3b4ff3e6-74d0-443a-b28a-3a83b971fcd3-kube-api-access-b6kjq\") pod \"odh-model-controller-7998c67dc4-6ntnl\" (UID: \"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:13.139014 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.138878    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b4ff3e6-74d0-443a-b28a-3a83b971fcd3-cert\") pod \"odh-model-controller-7998c67dc4-6ntnl\" (UID: \"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:13.141460 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.141437    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b4ff3e6-74d0-443a-b28a-3a83b971fcd3-cert\") pod \"odh-model-controller-7998c67dc4-6ntnl\" (UID: \"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:13.148451 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.148422    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kjq\" (UniqueName: \"kubernetes.io/projected/3b4ff3e6-74d0-443a-b28a-3a83b971fcd3-kube-api-access-b6kjq\") pod \"odh-model-controller-7998c67dc4-6ntnl\" (UID: \"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3\") " pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:13.192053 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.192033    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx"]
Jul 07 01:06:13.194044 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:06:13.194013    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da35937_89d5_48eb_bcb2_ef3d38e15c02.slice/crio-d1ec8b867b05a9c732f6d3edca684bd20f9850d0972c3e353e963211ddb56173 WatchSource:0}: Error finding container d1ec8b867b05a9c732f6d3edca684bd20f9850d0972c3e353e963211ddb56173: Status 404 returned error can't find the container with id d1ec8b867b05a9c732f6d3edca684bd20f9850d0972c3e353e963211ddb56173
Jul 07 01:06:13.227433 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.227406    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:13.263512 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.263490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:13.395540 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.395520    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6ntnl"]
Jul 07 01:06:13.398149 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:06:13.398123    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4ff3e6_74d0_443a_b28a_3a83b971fcd3.slice/crio-8323f3888bf41357c027a57c80d1dcf77b20c3351dd29b4c7af5a58ab156097d WatchSource:0}: Error finding container 8323f3888bf41357c027a57c80d1dcf77b20c3351dd29b4c7af5a58ab156097d: Status 404 returned error can't find the container with id 8323f3888bf41357c027a57c80d1dcf77b20c3351dd29b4c7af5a58ab156097d
Jul 07 01:06:13.399757 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.399737    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 01:06:13.443490 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.443457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl" event={"ID":"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3","Type":"ContainerStarted","Data":"8323f3888bf41357c027a57c80d1dcf77b20c3351dd29b4c7af5a58ab156097d"}
Jul 07 01:06:13.444544 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.444523    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx" event={"ID":"8da35937-89d5-48eb-bcb2-ef3d38e15c02","Type":"ContainerStarted","Data":"d1ec8b867b05a9c732f6d3edca684bd20f9850d0972c3e353e963211ddb56173"}
Jul 07 01:06:13.691014 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.690941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:13.701697 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.698992    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"]
Jul 07 01:06:13.704269 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.704245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.706860 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.706843    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:06:13.706957 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.706879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 07 01:06:13.706957 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.706893    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kpwxb\""
Jul 07 01:06:13.707100 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.707081    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 07 01:06:13.707182 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.707168    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 07 01:06:13.721272 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.721182    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"]
Jul 07 01:06:13.743068 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.743045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.743159 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.743073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.743159 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.743092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ced655d-7716-425e-9efc-e9657cb46081-local-certs\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.743159 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.743107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.743159 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.743140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg7nh\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-kube-api-access-qg7nh\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.743299 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.743191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-cacerts\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.743299 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.743250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-istio-token\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.791285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.791263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:13.844513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.844494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-istio-token\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.844611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.844529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.844611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.844548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.844611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.844574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ced655d-7716-425e-9efc-e9657cb46081-local-certs\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.844611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.844596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.844866 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.844630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg7nh\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-kube-api-access-qg7nh\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.844866 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.844663    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-cacerts\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.845304 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.845275    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.847396 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.847370    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ced655d-7716-425e-9efc-e9657cb46081-local-certs\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.847596 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.847576    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.847727 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.847616    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-cacerts\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.847727 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.847621    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.861921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.861833    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-istio-token\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.862632 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.862608    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg7nh\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-kube-api-access-qg7nh\") pod \"istiod-openshift-gateway-7574459656-pcnng\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:13.897521 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:13.897481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:14.013403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:14.013351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:14.083452 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:14.083387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:14.166618 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:14.166586    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"]
Jul 07 01:06:14.169600 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:06:14.169570    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ced655d_7716_425e_9efc_e9657cb46081.slice/crio-ec46ebcf5f1799c712c51dba3255b85de3f3e8c4c4bd69ff48dcb4eb422b9364 WatchSource:0}: Error finding container ec46ebcf5f1799c712c51dba3255b85de3f3e8c4c4bd69ff48dcb4eb422b9364: Status 404 returned error can't find the container with id ec46ebcf5f1799c712c51dba3255b85de3f3e8c4c4bd69ff48dcb4eb422b9364
Jul 07 01:06:14.452310 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:14.452229    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng" event={"ID":"6ced655d-7716-425e-9efc-e9657cb46081","Type":"ContainerStarted","Data":"ec46ebcf5f1799c712c51dba3255b85de3f3e8c4c4bd69ff48dcb4eb422b9364"}
Jul 07 01:06:14.712514 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:14.712409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:18.423552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:18.423512    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:06:18.423820 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:18.423575    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:06:19.143488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.143453    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:19.243437 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.243409    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:19.474056 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.474027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl" event={"ID":"3b4ff3e6-74d0-443a-b28a-3a83b971fcd3","Type":"ContainerStarted","Data":"9e0fa3e733b2841e24fa4f54d0fc671a7089c8b6f3a59c2b50531e48327c485b"}
Jul 07 01:06:19.474473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.474105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:19.475517 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.475492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx" event={"ID":"8da35937-89d5-48eb-bcb2-ef3d38e15c02","Type":"ContainerStarted","Data":"ce12b166f90f4db70e18eecc88ba5c072c90de44e341bbc2dc3eb882c193ae18"}
Jul 07 01:06:19.476924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.476906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng" event={"ID":"6ced655d-7716-425e-9efc-e9657cb46081","Type":"ContainerStarted","Data":"9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff"}
Jul 07 01:06:19.477112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.477097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:19.478405 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.478386    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-pcnng container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 01:06:19.478497 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.478426    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng" podUID="6ced655d-7716-425e-9efc-e9657cb46081" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 01:06:19.493612 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.493563    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl" podStartSLOduration=2.46227907 podStartE2EDuration="7.493549155s" totalImagesPullingTime="5.031270085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:06:19.49337635 +0000 UTC m=+307.768367672" watchObservedRunningTime="2026-07-07 01:06:19.493549155 +0000 UTC m=+307.768540479"
Jul 07 01:06:19.509225 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.509187    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9b59ff655-j9mqx" podStartSLOduration=2.283944493 podStartE2EDuration="7.50917549s" totalImagesPullingTime="5.225230997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:06:19.508002756 +0000 UTC m=+307.782994079" watchObservedRunningTime="2026-07-07 01:06:19.50917549 +0000 UTC m=+307.784166812"
Jul 07 01:06:19.526366 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.526326    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng" podStartSLOduration=2.275057331 podStartE2EDuration="6.526313718s" totalImagesPullingTime="4.251256387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:06:19.525349398 +0000 UTC m=+307.800340724" watchObservedRunningTime="2026-07-07 01:06:19.526313718 +0000 UTC m=+307.801305038"
Jul 07 01:06:19.686717 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:19.686671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:20.006244 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:20.006211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:20.104202 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:20.104172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:20.480817 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:20.480791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:23.846579 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.846543    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:23.900822 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.900792    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"]
Jul 07 01:06:23.903955 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.903934    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:23.906666 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.906642    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 07 01:06:23.906796 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.906670    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jvgtb\""
Jul 07 01:06:23.913147 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.913121    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"]
Jul 07 01:06:23.916167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.916148    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:23.917657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.917636    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"]
Jul 07 01:06:23.918570 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.918554    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 07 01:06:23.918660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.918574    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m2tg8\""
Jul 07 01:06:23.935779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.935755    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"]
Jul 07 01:06:23.944705 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:23.944660    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:24.029055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.029027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72g8p\" (UniqueName: \"kubernetes.io/projected/047b48cc-4194-4c32-bfa0-75dfb2ec0b9f-kube-api-access-72g8p\") pod \"kserve-controller-manager-d6f4b65b8-gzghl\" (UID: \"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:24.029195 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.029071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/047b48cc-4194-4c32-bfa0-75dfb2ec0b9f-cert\") pod \"kserve-controller-manager-d6f4b65b8-gzghl\" (UID: \"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:24.029195 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.029109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7h2r\" (UniqueName: \"kubernetes.io/projected/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-kube-api-access-n7h2r\") pod \"llmisvc-controller-manager-777c6bd7d5-m7l6m\" (UID: \"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.029195 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.029154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m7l6m\" (UID: \"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.130255 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.130154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/047b48cc-4194-4c32-bfa0-75dfb2ec0b9f-cert\") pod \"kserve-controller-manager-d6f4b65b8-gzghl\" (UID: \"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:24.130255 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.130249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7h2r\" (UniqueName: \"kubernetes.io/projected/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-kube-api-access-n7h2r\") pod \"llmisvc-controller-manager-777c6bd7d5-m7l6m\" (UID: \"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.130497 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.130282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m7l6m\" (UID: \"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.130497 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.130342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72g8p\" (UniqueName: \"kubernetes.io/projected/047b48cc-4194-4c32-bfa0-75dfb2ec0b9f-kube-api-access-72g8p\") pod \"kserve-controller-manager-d6f4b65b8-gzghl\" (UID: \"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:24.130497 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:06:24.130473    2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 07 01:06:24.130645 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:06:24.130557    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-cert podName:76febb9f-b8a4-4fcc-94f7-1c96dda0e81f nodeName:}" failed. No retries permitted until 2026-07-07 01:06:24.630532938 +0000 UTC m=+312.905524241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-cert") pod "llmisvc-controller-manager-777c6bd7d5-m7l6m" (UID: "76febb9f-b8a4-4fcc-94f7-1c96dda0e81f") : secret "llmisvc-webhook-server-cert" not found
Jul 07 01:06:24.133134 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.133109    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/047b48cc-4194-4c32-bfa0-75dfb2ec0b9f-cert\") pod \"kserve-controller-manager-d6f4b65b8-gzghl\" (UID: \"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:24.141884 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.141854    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72g8p\" (UniqueName: \"kubernetes.io/projected/047b48cc-4194-4c32-bfa0-75dfb2ec0b9f-kube-api-access-72g8p\") pod \"kserve-controller-manager-d6f4b65b8-gzghl\" (UID: \"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:24.142491 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.142471    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7h2r\" (UniqueName: \"kubernetes.io/projected/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-kube-api-access-n7h2r\") pod \"llmisvc-controller-manager-777c6bd7d5-m7l6m\" (UID: \"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.170549 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.170523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:24.214855 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.214823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:24.282228 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.282156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:24.353198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.353090    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"]
Jul 07 01:06:24.355770 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:06:24.355743    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047b48cc_4194_4c32_bfa0_75dfb2ec0b9f.slice/crio-d41a51c0cdb67e8428d529a5964e86b2dbb2a8911651579e61e0000d74b473b8 WatchSource:0}: Error finding container d41a51c0cdb67e8428d529a5964e86b2dbb2a8911651579e61e0000d74b473b8: Status 404 returned error can't find the container with id d41a51c0cdb67e8428d529a5964e86b2dbb2a8911651579e61e0000d74b473b8
Jul 07 01:06:24.500837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.500798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl" event={"ID":"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f","Type":"ContainerStarted","Data":"d41a51c0cdb67e8428d529a5964e86b2dbb2a8911651579e61e0000d74b473b8"}
Jul 07 01:06:24.633289 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.633250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m7l6m\" (UID: \"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.635803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.635786    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76febb9f-b8a4-4fcc-94f7-1c96dda0e81f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m7l6m\" (UID: \"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.828220 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.828128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:24.960619 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:24.960590    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"]
Jul 07 01:06:24.964604 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:06:24.964573    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod76febb9f_b8a4_4fcc_94f7_1c96dda0e81f.slice/crio-f4fe9e1badeab4a73a73a995370551f2d720c4bd9a799eb431bd568777302611 WatchSource:0}: Error finding container f4fe9e1badeab4a73a73a995370551f2d720c4bd9a799eb431bd568777302611: Status 404 returned error can't find the container with id f4fe9e1badeab4a73a73a995370551f2d720c4bd9a799eb431bd568777302611
Jul 07 01:06:25.110219 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:25.110132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:25.217576 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:25.217532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:25.379552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:25.379456    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:25.508279 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:25.508242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m" event={"ID":"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f","Type":"ContainerStarted","Data":"f4fe9e1badeab4a73a73a995370551f2d720c4bd9a799eb431bd568777302611"}
Jul 07 01:06:27.520737 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:27.520702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl" event={"ID":"047b48cc-4194-4c32-bfa0-75dfb2ec0b9f","Type":"ContainerStarted","Data":"2690f4d89659eae4421e35fa8ecf29409a95db169ba1e619c67497caf5d88b24"}
Jul 07 01:06:27.521232 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:27.520821    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:27.536248 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:27.536204    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl" podStartSLOduration=1.572672373 podStartE2EDuration="4.536188272s" totalImagesPullingTime="2.963515899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:06:27.535570802 +0000 UTC m=+315.810562126" watchObservedRunningTime="2026-07-07 01:06:27.536188272 +0000 UTC m=+315.811179597"
Jul 07 01:06:28.116659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:28.116622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:28.205091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:28.205058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:28.530023 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:28.529980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m" event={"ID":"76febb9f-b8a4-4fcc-94f7-1c96dda0e81f","Type":"ContainerStarted","Data":"2043a3dcee4075c044c83a8044ee18b4dd9eb5b5f6e6fec5f0377944ca5ce4a2"}
Jul 07 01:06:28.530401 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:28.530225    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:06:28.548518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:28.548476    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m" podStartSLOduration=2.262644118 podStartE2EDuration="5.548463205s" totalImagesPullingTime="3.285819087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:06:28.547608688 +0000 UTC m=+316.822600027" watchObservedRunningTime="2026-07-07 01:06:28.548463205 +0000 UTC m=+316.823454527"
Jul 07 01:06:29.734611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:29.734576    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"]
Jul 07 01:06:29.735022 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:29.734761    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng" podUID="6ced655d-7716-425e-9efc-e9657cb46081" containerName="discovery" containerID="cri-o://9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff" gracePeriod=30
Jul 07 01:06:30.001781 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.001755    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:30.085256 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085224    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-ca-configmap\") pod \"6ced655d-7716-425e-9efc-e9657cb46081\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") "
Jul 07 01:06:30.085256 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085257    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ced655d-7716-425e-9efc-e9657cb46081-local-certs\") pod \"6ced655d-7716-425e-9efc-e9657cb46081\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") "
Jul 07 01:06:30.085428 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085292    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-cacerts\") pod \"6ced655d-7716-425e-9efc-e9657cb46081\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") "
Jul 07 01:06:30.085533 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085502    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-dns-cert\") pod \"6ced655d-7716-425e-9efc-e9657cb46081\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") "
Jul 07 01:06:30.085652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085549    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg7nh\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-kube-api-access-qg7nh\") pod \"6ced655d-7716-425e-9efc-e9657cb46081\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") "
Jul 07 01:06:30.085652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085572    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6ced655d-7716-425e-9efc-e9657cb46081" (UID: "6ced655d-7716-425e-9efc-e9657cb46081"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:06:30.085652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-istio-token\") pod \"6ced655d-7716-425e-9efc-e9657cb46081\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") "
Jul 07 01:06:30.085652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-kubeconfig\") pod \"6ced655d-7716-425e-9efc-e9657cb46081\" (UID: \"6ced655d-7716-425e-9efc-e9657cb46081\") "
Jul 07 01:06:30.085908 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.085890    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-ca-configmap\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:06:30.088059 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.088030    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-cacerts" (OuterVolumeSpecName: "cacerts") pod "6ced655d-7716-425e-9efc-e9657cb46081" (UID: "6ced655d-7716-425e-9efc-e9657cb46081"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:06:30.088059 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.088041    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6ced655d-7716-425e-9efc-e9657cb46081" (UID: "6ced655d-7716-425e-9efc-e9657cb46081"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:06:30.088216 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.088076    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ced655d-7716-425e-9efc-e9657cb46081-local-certs" (OuterVolumeSpecName: "local-certs") pod "6ced655d-7716-425e-9efc-e9657cb46081" (UID: "6ced655d-7716-425e-9efc-e9657cb46081"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:06:30.088216 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.088101    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ced655d-7716-425e-9efc-e9657cb46081" (UID: "6ced655d-7716-425e-9efc-e9657cb46081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:06:30.088308 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.088237    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-kube-api-access-qg7nh" (OuterVolumeSpecName: "kube-api-access-qg7nh") pod "6ced655d-7716-425e-9efc-e9657cb46081" (UID: "6ced655d-7716-425e-9efc-e9657cb46081"). InnerVolumeSpecName "kube-api-access-qg7nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:06:30.088397 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.088374    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6ced655d-7716-425e-9efc-e9657cb46081" (UID: "6ced655d-7716-425e-9efc-e9657cb46081"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:06:30.186814 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.186788    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-kubeconfig\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:06:30.186814 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.186810    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ced655d-7716-425e-9efc-e9657cb46081-local-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:06:30.186814 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.186818    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-cacerts\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:06:30.187017 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.186826    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6ced655d-7716-425e-9efc-e9657cb46081-istio-csr-dns-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:06:30.187017 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.186835    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg7nh\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-kube-api-access-qg7nh\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:06:30.187017 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.186844    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ced655d-7716-425e-9efc-e9657cb46081-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:06:30.482643 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.482624    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6ntnl"
Jul 07 01:06:30.540632 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.540599    2562 generic.go:356] "Generic (PLEG): container finished" podID="6ced655d-7716-425e-9efc-e9657cb46081" containerID="9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff" exitCode=0
Jul 07 01:06:30.540805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.540642    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"
Jul 07 01:06:30.540805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.540676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng" event={"ID":"6ced655d-7716-425e-9efc-e9657cb46081","Type":"ContainerDied","Data":"9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff"}
Jul 07 01:06:30.540805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.540754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-pcnng" event={"ID":"6ced655d-7716-425e-9efc-e9657cb46081","Type":"ContainerDied","Data":"ec46ebcf5f1799c712c51dba3255b85de3f3e8c4c4bd69ff48dcb4eb422b9364"}
Jul 07 01:06:30.540805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.540790    2562 scope.go:117] "RemoveContainer" containerID="9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff"
Jul 07 01:06:30.554886 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.554869    2562 scope.go:117] "RemoveContainer" containerID="9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff"
Jul 07 01:06:30.555193 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:06:30.555173    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff\": container with ID starting with 9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff not found: ID does not exist" containerID="9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff"
Jul 07 01:06:30.555272 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.555199    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff"} err="failed to get container status \"9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff\": rpc error: code = NotFound desc = could not find container \"9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff\": container with ID starting with 9e09afd324d73b98c49bc56c972d54a37e7899b98c37f743d51192ffe68dadff not found: ID does not exist"
Jul 07 01:06:30.558399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.558378    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"]
Jul 07 01:06:30.561941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:30.561919    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-pcnng"]
Jul 07 01:06:32.288360 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:32.288324    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ced655d-7716-425e-9efc-e9657cb46081" path="/var/lib/kubelet/pods/6ced655d-7716-425e-9efc-e9657cb46081/volumes"
Jul 07 01:06:52.341141 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:52.341106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:06:58.535164 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:58.535134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gzghl"
Jul 07 01:06:59.536653 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:06:59.536625    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m7l6m"
Jul 07 01:07:13.601929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.601875    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:13.701058 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.701027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:13.740374 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.740344    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-49vcw"]
Jul 07 01:07:13.740616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.740604    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ced655d-7716-425e-9efc-e9657cb46081" containerName="discovery"
Jul 07 01:07:13.740657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.740617    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ced655d-7716-425e-9efc-e9657cb46081" containerName="discovery"
Jul 07 01:07:13.740764 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.740708    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ced655d-7716-425e-9efc-e9657cb46081" containerName="discovery"
Jul 07 01:07:13.744996 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.744971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-49vcw"
Jul 07 01:07:13.747832 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.747806    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 01:07:13.747952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.747836    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 01:07:13.748830 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.748806    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-727vj\""
Jul 07 01:07:13.754115 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.754094    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-49vcw"]
Jul 07 01:07:13.787832 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.787807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:13.804067 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.804043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xb8\" (UniqueName: \"kubernetes.io/projected/e7271e29-7e4a-4ad8-bdc8-38825af0361d-kube-api-access-99xb8\") pod \"kuadrant-operator-catalog-49vcw\" (UID: \"e7271e29-7e4a-4ad8-bdc8-38825af0361d\") " pod="kuadrant-system/kuadrant-operator-catalog-49vcw"
Jul 07 01:07:13.904939 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.904881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xb8\" (UniqueName: \"kubernetes.io/projected/e7271e29-7e4a-4ad8-bdc8-38825af0361d-kube-api-access-99xb8\") pod \"kuadrant-operator-catalog-49vcw\" (UID: \"e7271e29-7e4a-4ad8-bdc8-38825af0361d\") " pod="kuadrant-system/kuadrant-operator-catalog-49vcw"
Jul 07 01:07:13.912591 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:13.912572    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xb8\" (UniqueName: \"kubernetes.io/projected/e7271e29-7e4a-4ad8-bdc8-38825af0361d-kube-api-access-99xb8\") pod \"kuadrant-operator-catalog-49vcw\" (UID: \"e7271e29-7e4a-4ad8-bdc8-38825af0361d\") " pod="kuadrant-system/kuadrant-operator-catalog-49vcw"
Jul 07 01:07:14.056116 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.056092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-49vcw"
Jul 07 01:07:14.105867 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.105816    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-49vcw"]
Jul 07 01:07:14.185040 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.185017    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-49vcw"]
Jul 07 01:07:14.187561 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:07:14.187534    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7271e29_7e4a_4ad8_bdc8_38825af0361d.slice/crio-04dbacf74b00871342bcbdd66e1c6916f4810a573ea5643fc71fd259d1e957b0 WatchSource:0}: Error finding container 04dbacf74b00871342bcbdd66e1c6916f4810a573ea5643fc71fd259d1e957b0: Status 404 returned error can't find the container with id 04dbacf74b00871342bcbdd66e1c6916f4810a573ea5643fc71fd259d1e957b0
Jul 07 01:07:14.314903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.314874    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wgzmx"]
Jul 07 01:07:14.319001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.318985    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:14.326233 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.326201    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wgzmx"]
Jul 07 01:07:14.409270 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.409244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fqmm\" (UniqueName: \"kubernetes.io/projected/41ee820d-3002-407c-a183-6bf968208593-kube-api-access-7fqmm\") pod \"kuadrant-operator-catalog-wgzmx\" (UID: \"41ee820d-3002-407c-a183-6bf968208593\") " pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:14.510304 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.510282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fqmm\" (UniqueName: \"kubernetes.io/projected/41ee820d-3002-407c-a183-6bf968208593-kube-api-access-7fqmm\") pod \"kuadrant-operator-catalog-wgzmx\" (UID: \"41ee820d-3002-407c-a183-6bf968208593\") " pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:14.518127 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.518101    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fqmm\" (UniqueName: \"kubernetes.io/projected/41ee820d-3002-407c-a183-6bf968208593-kube-api-access-7fqmm\") pod \"kuadrant-operator-catalog-wgzmx\" (UID: \"41ee820d-3002-407c-a183-6bf968208593\") " pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:14.537080 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.537059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:14.628538 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.628512    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:14.718051 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.718020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-49vcw" event={"ID":"e7271e29-7e4a-4ad8-bdc8-38825af0361d","Type":"ContainerStarted","Data":"04dbacf74b00871342bcbdd66e1c6916f4810a573ea5643fc71fd259d1e957b0"}
Jul 07 01:07:14.746473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.746417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:14.777636 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:14.777608    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wgzmx"]
Jul 07 01:07:14.800659 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:07:14.800628    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ee820d_3002_407c_a183_6bf968208593.slice/crio-ce61b820a9c37ea031ce00861bac4ad4ceb6f719ea7ce190573c2b5b119e7597 WatchSource:0}: Error finding container ce61b820a9c37ea031ce00861bac4ad4ceb6f719ea7ce190573c2b5b119e7597: Status 404 returned error can't find the container with id ce61b820a9c37ea031ce00861bac4ad4ceb6f719ea7ce190573c2b5b119e7597
Jul 07 01:07:15.724405 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:15.724366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx" event={"ID":"41ee820d-3002-407c-a183-6bf968208593","Type":"ContainerStarted","Data":"ce61b820a9c37ea031ce00861bac4ad4ceb6f719ea7ce190573c2b5b119e7597"}
Jul 07 01:07:16.729577 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:16.729545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx" event={"ID":"41ee820d-3002-407c-a183-6bf968208593","Type":"ContainerStarted","Data":"6b7b5ad09f07e4215146263eb06c045cce853fea26bde6af73fa77d535e5735f"}
Jul 07 01:07:16.731019 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:16.730985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-49vcw" event={"ID":"e7271e29-7e4a-4ad8-bdc8-38825af0361d","Type":"ContainerStarted","Data":"f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37"}
Jul 07 01:07:16.731155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:16.731036    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-49vcw" podUID="e7271e29-7e4a-4ad8-bdc8-38825af0361d" containerName="registry-server" containerID="cri-o://f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37" gracePeriod=2
Jul 07 01:07:16.743484 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:16.743437    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx" podStartSLOduration=1.234636181 podStartE2EDuration="2.743423165s" totalImagesPullingTime="1.508786984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:07:16.742830878 +0000 UTC m=+365.017822199" watchObservedRunningTime="2026-07-07 01:07:16.743423165 +0000 UTC m=+365.018414490"
Jul 07 01:07:16.757233 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:16.757199    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-49vcw" podStartSLOduration=1.638177918 podStartE2EDuration="3.757185869s" totalImagesPullingTime="2.119007951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:07:16.755790032 +0000 UTC m=+365.030781353" watchObservedRunningTime="2026-07-07 01:07:16.757185869 +0000 UTC m=+365.032177190"
Jul 07 01:07:16.970083 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:16.970054    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-49vcw"
Jul 07 01:07:17.032301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.032240    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99xb8\" (UniqueName: \"kubernetes.io/projected/e7271e29-7e4a-4ad8-bdc8-38825af0361d-kube-api-access-99xb8\") pod \"e7271e29-7e4a-4ad8-bdc8-38825af0361d\" (UID: \"e7271e29-7e4a-4ad8-bdc8-38825af0361d\") "
Jul 07 01:07:17.034387 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.034366    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7271e29-7e4a-4ad8-bdc8-38825af0361d-kube-api-access-99xb8" (OuterVolumeSpecName: "kube-api-access-99xb8") pod "e7271e29-7e4a-4ad8-bdc8-38825af0361d" (UID: "e7271e29-7e4a-4ad8-bdc8-38825af0361d"). InnerVolumeSpecName "kube-api-access-99xb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:07:17.132837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.132800    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xb8\" (UniqueName: \"kubernetes.io/projected/e7271e29-7e4a-4ad8-bdc8-38825af0361d-kube-api-access-99xb8\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:07:17.736884 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.736852    2562 generic.go:356] "Generic (PLEG): container finished" podID="e7271e29-7e4a-4ad8-bdc8-38825af0361d" containerID="f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37" exitCode=0
Jul 07 01:07:17.737268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.736901    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-49vcw"
Jul 07 01:07:17.737268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.736937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-49vcw" event={"ID":"e7271e29-7e4a-4ad8-bdc8-38825af0361d","Type":"ContainerDied","Data":"f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37"}
Jul 07 01:07:17.737268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.736993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-49vcw" event={"ID":"e7271e29-7e4a-4ad8-bdc8-38825af0361d","Type":"ContainerDied","Data":"04dbacf74b00871342bcbdd66e1c6916f4810a573ea5643fc71fd259d1e957b0"}
Jul 07 01:07:17.737268 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.737014    2562 scope.go:117] "RemoveContainer" containerID="f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37"
Jul 07 01:07:17.746936 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.746918    2562 scope.go:117] "RemoveContainer" containerID="f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37"
Jul 07 01:07:17.747230 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:07:17.747210    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37\": container with ID starting with f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37 not found: ID does not exist" containerID="f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37"
Jul 07 01:07:17.747292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.747240    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37"} err="failed to get container status \"f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37\": rpc error: code = NotFound desc = could not find container \"f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37\": container with ID starting with f6167adc72c1c5008b082edbdbc89b92b456ea4fc82afc853dd897873a063c37 not found: ID does not exist"
Jul 07 01:07:17.756373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.756351    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-49vcw"]
Jul 07 01:07:17.761581 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:17.761557    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-49vcw"]
Jul 07 01:07:18.289163 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:18.289128    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7271e29-7e4a-4ad8-bdc8-38825af0361d" path="/var/lib/kubelet/pods/e7271e29-7e4a-4ad8-bdc8-38825af0361d/volumes"
Jul 07 01:07:24.628718 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:24.628673    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:24.629116 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:24.628928    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:24.653495 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:24.653470    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:24.788710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:24.788668    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wgzmx"
Jul 07 01:07:34.797909 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:34.797877    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:37.307654 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.307621    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"]
Jul 07 01:07:37.308044 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.307926    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7271e29-7e4a-4ad8-bdc8-38825af0361d" containerName="registry-server"
Jul 07 01:07:37.308044 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.307938    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7271e29-7e4a-4ad8-bdc8-38825af0361d" containerName="registry-server"
Jul 07 01:07:37.308044 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.307985    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7271e29-7e4a-4ad8-bdc8-38825af0361d" containerName="registry-server"
Jul 07 01:07:37.310866 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.310851    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:07:37.313257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.313235    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-r246x\""
Jul 07 01:07:37.385280 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.385245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xxq\" (UniqueName: \"kubernetes.io/projected/bd984a03-5cc4-4b56-b490-4e102ed26f9a-kube-api-access-z7xxq\") pod \"limitador-operator-controller-manager-859d7b78bd-88mwl\" (UID: \"bd984a03-5cc4-4b56-b490-4e102ed26f9a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:07:37.486445 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.486399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xxq\" (UniqueName: \"kubernetes.io/projected/bd984a03-5cc4-4b56-b490-4e102ed26f9a-kube-api-access-z7xxq\") pod \"limitador-operator-controller-manager-859d7b78bd-88mwl\" (UID: \"bd984a03-5cc4-4b56-b490-4e102ed26f9a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:07:37.497175 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.497145    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xxq\" (UniqueName: \"kubernetes.io/projected/bd984a03-5cc4-4b56-b490-4e102ed26f9a-kube-api-access-z7xxq\") pod \"limitador-operator-controller-manager-859d7b78bd-88mwl\" (UID: \"bd984a03-5cc4-4b56-b490-4e102ed26f9a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:07:37.620637 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:37.620551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:07:38.208616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:38.208580    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:38.325991 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:38.325929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:40.660746 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:40.660664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:40.760650 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:40.760617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:42.624188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.624145    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"]
Jul 07 01:07:42.631247 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.631215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:42.631500 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.631479    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:42.634759 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.634740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dvc5k\""
Jul 07 01:07:42.729484 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.729456    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:42.733399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.733380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbh2p\" (UniqueName: \"kubernetes.io/projected/48efc560-6e55-4eac-a5b1-0999934fdc85-kube-api-access-hbh2p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:42.733500 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.733416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48efc560-6e55-4eac-a5b1-0999934fdc85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:42.834264 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.834233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbh2p\" (UniqueName: \"kubernetes.io/projected/48efc560-6e55-4eac-a5b1-0999934fdc85-kube-api-access-hbh2p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:42.834392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.834287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48efc560-6e55-4eac-a5b1-0999934fdc85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:42.834741 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.834716    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48efc560-6e55-4eac-a5b1-0999934fdc85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:42.842836 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.842812    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbh2p\" (UniqueName: \"kubernetes.io/projected/48efc560-6e55-4eac-a5b1-0999934fdc85-kube-api-access-hbh2p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:42.942570 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:42.942502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:07:43.417168 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:43.417106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:43.700148 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:43.700065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:44.782056 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:44.782019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:44.881829 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:44.881791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:45.398406 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.398370    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"]
Jul 07 01:07:45.404777 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.404754    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:07:45.407875 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.407852    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 07 01:07:45.408018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.407999    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-22l5r\""
Jul 07 01:07:45.555413 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.555384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn98z\" (UniqueName: \"kubernetes.io/projected/95b615bf-4983-4ded-a9df-4e1b999314b1-kube-api-access-tn98z\") pod \"dns-operator-controller-manager-864dd78c4c-t9vd5\" (UID: \"95b615bf-4983-4ded-a9df-4e1b999314b1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:07:45.655947 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.655875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn98z\" (UniqueName: \"kubernetes.io/projected/95b615bf-4983-4ded-a9df-4e1b999314b1-kube-api-access-tn98z\") pod \"dns-operator-controller-manager-864dd78c4c-t9vd5\" (UID: \"95b615bf-4983-4ded-a9df-4e1b999314b1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:07:45.665924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.665896    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn98z\" (UniqueName: \"kubernetes.io/projected/95b615bf-4983-4ded-a9df-4e1b999314b1-kube-api-access-tn98z\") pod \"dns-operator-controller-manager-864dd78c4c-t9vd5\" (UID: \"95b615bf-4983-4ded-a9df-4e1b999314b1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:07:45.716750 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.716724    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:07:45.803141 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:45.803106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:46.034347 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:46.034321    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:46.193582 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:46.193525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:48.432613 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:48.432579    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:48.533561 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:48.533530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:48.810823 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:48.810790    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"]
Jul 07 01:07:48.816543 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:48.816521    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"
Jul 07 01:07:48.819139 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:48.819122    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lzkr6\""
Jul 07 01:07:48.979994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:48.979968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthg6\" (UniqueName: \"kubernetes.io/projected/9458d5fb-093b-49f9-90f2-8fa551735d97-kube-api-access-jthg6\") pod \"authorino-operator-75cc8d5cb7-xgbx9\" (UID: \"9458d5fb-093b-49f9-90f2-8fa551735d97\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"
Jul 07 01:07:49.080542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:49.080483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthg6\" (UniqueName: \"kubernetes.io/projected/9458d5fb-093b-49f9-90f2-8fa551735d97-kube-api-access-jthg6\") pod \"authorino-operator-75cc8d5cb7-xgbx9\" (UID: \"9458d5fb-093b-49f9-90f2-8fa551735d97\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"
Jul 07 01:07:49.092074 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:49.092045    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthg6\" (UniqueName: \"kubernetes.io/projected/9458d5fb-093b-49f9-90f2-8fa551735d97-kube-api-access-jthg6\") pod \"authorino-operator-75cc8d5cb7-xgbx9\" (UID: \"9458d5fb-093b-49f9-90f2-8fa551735d97\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"
Jul 07 01:07:49.127098 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:49.127077    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"
Jul 07 01:07:50.405319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:50.405278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:50.505629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:50.505592    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:53.599803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:53.599758    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:53.664757 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:53.664708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:59.532530 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:59.532496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:59.593520 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:59.593465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:59.846175 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:59.846082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:07:59.930661 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:07:59.930633    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:06.417162 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:06.417119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:06.515566 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:06.515532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:09.022753 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:09.022717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:09.123570 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:09.123535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:10.761526 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:10.761494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:10.821244 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:10.821207    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:11.041128 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:11.041040    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:11.103833 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:11.103790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:12.609971 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:12.609931    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:12.674301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:12.674249    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:17.720183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:17.720143    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"]
Jul 07 01:08:17.723827 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:17.723795    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"]
Jul 07 01:08:17.727609 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:17.727584    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"]
Jul 07 01:08:17.732988 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:17.732949    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"]
Jul 07 01:08:17.918312 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:17.917297    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"]
Jul 07 01:08:17.920509 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:17.920478    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9458d5fb_093b_49f9_90f2_8fa551735d97.slice/crio-3a4ed3826fafa3b7424fc8b8ccad086388f5aff46000c7aefa98a2b06e701140 WatchSource:0}: Error finding container 3a4ed3826fafa3b7424fc8b8ccad086388f5aff46000c7aefa98a2b06e701140: Status 404 returned error can't find the container with id 3a4ed3826fafa3b7424fc8b8ccad086388f5aff46000c7aefa98a2b06e701140
Jul 07 01:08:17.988507 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:17.988438    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9" event={"ID":"9458d5fb-093b-49f9-90f2-8fa551735d97","Type":"ContainerStarted","Data":"3a4ed3826fafa3b7424fc8b8ccad086388f5aff46000c7aefa98a2b06e701140"}
Jul 07 01:08:18.147629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:18.147599    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"]
Jul 07 01:08:18.150953 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:18.150906    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"]
Jul 07 01:08:18.151757 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:18.151735    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd984a03_5cc4_4b56_b490_4e102ed26f9a.slice/crio-0ccf1deb19309e1319a24f17dab262892b12bbd66238fefb43a815089288c29f WatchSource:0}: Error finding container 0ccf1deb19309e1319a24f17dab262892b12bbd66238fefb43a815089288c29f: Status 404 returned error can't find the container with id 0ccf1deb19309e1319a24f17dab262892b12bbd66238fefb43a815089288c29f
Jul 07 01:08:18.152561 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:18.152538    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"]
Jul 07 01:08:18.153406 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:18.153381    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48efc560_6e55_4eac_a5b1_0999934fdc85.slice/crio-55c316c3541d44aedfb64031ce83d7387af760a9f74b0730a0a313f933e7f28e WatchSource:0}: Error finding container 55c316c3541d44aedfb64031ce83d7387af760a9f74b0730a0a313f933e7f28e: Status 404 returned error can't find the container with id 55c316c3541d44aedfb64031ce83d7387af760a9f74b0730a0a313f933e7f28e
Jul 07 01:08:18.154181 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:18.154154    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b615bf_4983_4ded_a9df_4e1b999314b1.slice/crio-5f310f4d12b72fdd969a0e01be88977e7da2339a971c7a04d0217de86df3343f WatchSource:0}: Error finding container 5f310f4d12b72fdd969a0e01be88977e7da2339a971c7a04d0217de86df3343f: Status 404 returned error can't find the container with id 5f310f4d12b72fdd969a0e01be88977e7da2339a971c7a04d0217de86df3343f
Jul 07 01:08:18.998228 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:18.998151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5" event={"ID":"95b615bf-4983-4ded-a9df-4e1b999314b1","Type":"ContainerStarted","Data":"5f310f4d12b72fdd969a0e01be88977e7da2339a971c7a04d0217de86df3343f"}
Jul 07 01:08:19.000663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:19.000631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc" event={"ID":"48efc560-6e55-4eac-a5b1-0999934fdc85","Type":"ContainerStarted","Data":"55c316c3541d44aedfb64031ce83d7387af760a9f74b0730a0a313f933e7f28e"}
Jul 07 01:08:19.006772 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:19.006743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl" event={"ID":"bd984a03-5cc4-4b56-b490-4e102ed26f9a","Type":"ContainerStarted","Data":"0ccf1deb19309e1319a24f17dab262892b12bbd66238fefb43a815089288c29f"}
Jul 07 01:08:20.215425 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:20.215355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:20.272097 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:20.272062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:21.646492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:21.646462    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 07 01:08:22.022943 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.022915    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5" event={"ID":"95b615bf-4983-4ded-a9df-4e1b999314b1","Type":"ContainerStarted","Data":"831a72422eb7239ebaffa3706f595f7eb7ea056ff22dadae778c336c3af6d8c8"}
Jul 07 01:08:22.023116 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.023100    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:08:22.024530 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.024503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl" event={"ID":"bd984a03-5cc4-4b56-b490-4e102ed26f9a","Type":"ContainerStarted","Data":"cb74db967f42a868e44df2dc442a07bc8c27831ab16ca19f33dc150cb740c532"}
Jul 07 01:08:22.024654 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.024572    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:22.026108 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.026082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9" event={"ID":"9458d5fb-093b-49f9-90f2-8fa551735d97","Type":"ContainerStarted","Data":"f4d15d1ff5777121eba5c48df3bb8e16d2efaff5f31deb0d00083e28989647df"}
Jul 07 01:08:22.026237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.026147    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"
Jul 07 01:08:22.041503 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.041454    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5" podStartSLOduration=33.553822153 podStartE2EDuration="37.04144222s" totalImagesPullingTime="3.487620067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:22.040189006 +0000 UTC m=+430.315180341" watchObservedRunningTime="2026-07-07 01:08:22.04144222 +0000 UTC m=+430.316433539"
Jul 07 01:08:22.055907 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.055867    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9" podStartSLOduration=31.104774296 podStartE2EDuration="34.055853664s" totalImagesPullingTime="2.951079368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:22.055040983 +0000 UTC m=+430.330032344" watchObservedRunningTime="2026-07-07 01:08:22.055853664 +0000 UTC m=+430.330844986"
Jul 07 01:08:22.072305 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:22.072244    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl" podStartSLOduration=42.180686588 podStartE2EDuration="45.072231313s" totalImagesPullingTime="2.891544725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:22.070949766 +0000 UTC m=+430.345941091" watchObservedRunningTime="2026-07-07 01:08:22.072231313 +0000 UTC m=+430.347222635"
Jul 07 01:08:25.046002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:25.045961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc" event={"ID":"48efc560-6e55-4eac-a5b1-0999934fdc85","Type":"ContainerStarted","Data":"61c63de4f53d5878ca5d712c61f61fce700be92f3ec9eb69a59e3e3293e7ede6"}
Jul 07 01:08:25.046488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:25.046123    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:25.066289 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:25.066247    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc" podStartSLOduration=36.352398608 podStartE2EDuration="43.066235745s" totalImagesPullingTime="6.713837137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:25.064353078 +0000 UTC m=+433.339344400" watchObservedRunningTime="2026-07-07 01:08:25.066235745 +0000 UTC m=+433.341227066"
Jul 07 01:08:26.023893 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:26.023859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:26.123925 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:26.123878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:27.618183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.618149    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"]
Jul 07 01:08:27.621630 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.621609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.622170 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.622147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:27.624106 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.624084    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 07 01:08:27.624205 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.624092    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 07 01:08:27.624205 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.624094    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mmqgp\""
Jul 07 01:08:27.628927 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.628745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"]
Jul 07 01:08:27.669698 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.669663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/797adc9d-d61e-45fa-83b3-5e901b21d407-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.669785 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.669713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/797adc9d-d61e-45fa-83b3-5e901b21d407-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.669785 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.669768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ng8\" (UniqueName: \"kubernetes.io/projected/797adc9d-d61e-45fa-83b3-5e901b21d407-kube-api-access-n2ng8\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.770869 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.770834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/797adc9d-d61e-45fa-83b3-5e901b21d407-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.770869 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.770873    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/797adc9d-d61e-45fa-83b3-5e901b21d407-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.771111 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.770909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ng8\" (UniqueName: \"kubernetes.io/projected/797adc9d-d61e-45fa-83b3-5e901b21d407-kube-api-access-n2ng8\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.771111 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:27.771005    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 07 01:08:27.771111 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:27.771085    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/797adc9d-d61e-45fa-83b3-5e901b21d407-plugin-serving-cert podName:797adc9d-d61e-45fa-83b3-5e901b21d407 nodeName:}" failed. No retries permitted until 2026-07-07 01:08:28.271062723 +0000 UTC m=+436.546054039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/797adc9d-d61e-45fa-83b3-5e901b21d407-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-z58vn" (UID: "797adc9d-d61e-45fa-83b3-5e901b21d407") : secret "plugin-serving-cert" not found
Jul 07 01:08:27.771543 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.771523    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/797adc9d-d61e-45fa-83b3-5e901b21d407-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.782324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.782302    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ng8\" (UniqueName: \"kubernetes.io/projected/797adc9d-d61e-45fa-83b3-5e901b21d407-kube-api-access-n2ng8\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:27.927260 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:27.927171    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:08:28.274871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:28.274823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/797adc9d-d61e-45fa-83b3-5e901b21d407-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:28.277377 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:28.277349    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/797adc9d-d61e-45fa-83b3-5e901b21d407-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z58vn\" (UID: \"797adc9d-d61e-45fa-83b3-5e901b21d407\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:28.531707 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:28.531612    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"
Jul 07 01:08:28.666383 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:28.666362    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn"]
Jul 07 01:08:28.668770 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:28.668743    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod797adc9d_d61e_45fa_83b3_5e901b21d407.slice/crio-6d0d94553d6a8ea001f99ebb443655b044d611449ede5697f2e77cf12472c356 WatchSource:0}: Error finding container 6d0d94553d6a8ea001f99ebb443655b044d611449ede5697f2e77cf12472c356: Status 404 returned error can't find the container with id 6d0d94553d6a8ea001f99ebb443655b044d611449ede5697f2e77cf12472c356
Jul 07 01:08:29.064069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:29.064033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn" event={"ID":"797adc9d-d61e-45fa-83b3-5e901b21d407","Type":"ContainerStarted","Data":"6d0d94553d6a8ea001f99ebb443655b044d611449ede5697f2e77cf12472c356"}
Jul 07 01:08:33.032384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:33.032239    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:33.032814 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:33.032422    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xgbx9"
Jul 07 01:08:33.032814 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:33.032579    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:08:36.052924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:36.052844    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:37.870707 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.870643    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"]
Jul 07 01:08:37.871183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.870928    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" containerName="manager" containerID="cri-o://61c63de4f53d5878ca5d712c61f61fce700be92f3ec9eb69a59e3e3293e7ede6" gracePeriod=2
Jul 07 01:08:37.872665 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.872628    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"]
Jul 07 01:08:37.873036 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:37.873002    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:37.874861 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.874833    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"]
Jul 07 01:08:37.875244 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.875081    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" containerName="manager" containerID="cri-o://cb74db967f42a868e44df2dc442a07bc8c27831ab16ca19f33dc150cb740c532" gracePeriod=2
Jul 07 01:08:37.876408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.876385    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"]
Jul 07 01:08:37.877080 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:37.877052    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-88mwl\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:37.877576 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.877558    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"]
Jul 07 01:08:37.878126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.878105    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" containerName="manager"
Jul 07 01:08:37.878226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.878138    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" containerName="manager"
Jul 07 01:08:37.878226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.878168    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" containerName="manager"
Jul 07 01:08:37.878226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.878175    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" containerName="manager"
Jul 07 01:08:37.878372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.878252    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" containerName="manager"
Jul 07 01:08:37.878372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.878264    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" containerName="manager"
Jul 07 01:08:37.879267 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:37.879233    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:37.881836 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.881816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:37.883906 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:37.883882    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-88mwl\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:37.885772 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:37.885748    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:37.894266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.894243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"]
Jul 07 01:08:37.898199 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.898181    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"]
Jul 07 01:08:37.901530 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.901513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"
Jul 07 01:08:37.907549 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:37.907522    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-88mwl\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:37.911147 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:37.911122    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:37.915967 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.915946    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"]
Jul 07 01:08:37.957544 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.957502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8mj\" (UniqueName: \"kubernetes.io/projected/dc9ca5e9-8216-464e-83f5-50cdd6c580d4-kube-api-access-gh8mj\") pod \"limitador-operator-controller-manager-859d7b78bd-x8mn4\" (UID: \"dc9ca5e9-8216-464e-83f5-50cdd6c580d4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"
Jul 07 01:08:37.957663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.957558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6452d\" (UniqueName: \"kubernetes.io/projected/062f00db-6780-4cd1-b6c1-26561deaec56-kube-api-access-6452d\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-c29jc\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:37.957663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:37.957626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062f00db-6780-4cd1-b6c1-26561deaec56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-c29jc\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:38.058123 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.058096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062f00db-6780-4cd1-b6c1-26561deaec56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-c29jc\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:38.058244 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.058159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8mj\" (UniqueName: \"kubernetes.io/projected/dc9ca5e9-8216-464e-83f5-50cdd6c580d4-kube-api-access-gh8mj\") pod \"limitador-operator-controller-manager-859d7b78bd-x8mn4\" (UID: \"dc9ca5e9-8216-464e-83f5-50cdd6c580d4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"
Jul 07 01:08:38.058244 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.058195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6452d\" (UniqueName: \"kubernetes.io/projected/062f00db-6780-4cd1-b6c1-26561deaec56-kube-api-access-6452d\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-c29jc\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:38.058567 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.058532    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062f00db-6780-4cd1-b6c1-26561deaec56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-c29jc\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:38.077944 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.077900    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6452d\" (UniqueName: \"kubernetes.io/projected/062f00db-6780-4cd1-b6c1-26561deaec56-kube-api-access-6452d\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-c29jc\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:38.081645 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.081604    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8mj\" (UniqueName: \"kubernetes.io/projected/dc9ca5e9-8216-464e-83f5-50cdd6c580d4-kube-api-access-gh8mj\") pod \"limitador-operator-controller-manager-859d7b78bd-x8mn4\" (UID: \"dc9ca5e9-8216-464e-83f5-50cdd6c580d4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"
Jul 07 01:08:38.114740 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.114712    2562 generic.go:356] "Generic (PLEG): container finished" podID="48efc560-6e55-4eac-a5b1-0999934fdc85" containerID="61c63de4f53d5878ca5d712c61f61fce700be92f3ec9eb69a59e3e3293e7ede6" exitCode=0
Jul 07 01:08:38.116249 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.116214    2562 generic.go:356] "Generic (PLEG): container finished" podID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" containerID="cb74db967f42a868e44df2dc442a07bc8c27831ab16ca19f33dc150cb740c532" exitCode=0
Jul 07 01:08:38.126751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.126676    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:38.128793 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.128778    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:38.129419 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:38.129394    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:38.131198 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:38.131179    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-88mwl\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:38.133039 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:38.133021    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2z6hc\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:38.134850 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:08:38.134830    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-88mwl\" is forbidden: User \"system:node:ip-10-0-142-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-16.ec2.internal' and this object" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:38.159100 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.159081    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7xxq\" (UniqueName: \"kubernetes.io/projected/bd984a03-5cc4-4b56-b490-4e102ed26f9a-kube-api-access-z7xxq\") pod \"bd984a03-5cc4-4b56-b490-4e102ed26f9a\" (UID: \"bd984a03-5cc4-4b56-b490-4e102ed26f9a\") "
Jul 07 01:08:38.161063 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.161039    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd984a03-5cc4-4b56-b490-4e102ed26f9a-kube-api-access-z7xxq" (OuterVolumeSpecName: "kube-api-access-z7xxq") pod "bd984a03-5cc4-4b56-b490-4e102ed26f9a" (UID: "bd984a03-5cc4-4b56-b490-4e102ed26f9a"). InnerVolumeSpecName "kube-api-access-z7xxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:08:38.259611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.259586    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48efc560-6e55-4eac-a5b1-0999934fdc85-extensions-socket-volume\") pod \"48efc560-6e55-4eac-a5b1-0999934fdc85\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") "
Jul 07 01:08:38.259738 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.259615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbh2p\" (UniqueName: \"kubernetes.io/projected/48efc560-6e55-4eac-a5b1-0999934fdc85-kube-api-access-hbh2p\") pod \"48efc560-6e55-4eac-a5b1-0999934fdc85\" (UID: \"48efc560-6e55-4eac-a5b1-0999934fdc85\") "
Jul 07 01:08:38.259801 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.259766    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xxq\" (UniqueName: \"kubernetes.io/projected/bd984a03-5cc4-4b56-b490-4e102ed26f9a-kube-api-access-z7xxq\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:08:38.260123 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.260095    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48efc560-6e55-4eac-a5b1-0999934fdc85-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "48efc560-6e55-4eac-a5b1-0999934fdc85" (UID: "48efc560-6e55-4eac-a5b1-0999934fdc85"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:08:38.261630 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.261610    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48efc560-6e55-4eac-a5b1-0999934fdc85-kube-api-access-hbh2p" (OuterVolumeSpecName: "kube-api-access-hbh2p") pod "48efc560-6e55-4eac-a5b1-0999934fdc85" (UID: "48efc560-6e55-4eac-a5b1-0999934fdc85"). InnerVolumeSpecName "kube-api-access-hbh2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:08:38.288455 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.288432    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48efc560-6e55-4eac-a5b1-0999934fdc85" path="/var/lib/kubelet/pods/48efc560-6e55-4eac-a5b1-0999934fdc85/volumes"
Jul 07 01:08:38.288752 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.288739    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd984a03-5cc4-4b56-b490-4e102ed26f9a" path="/var/lib/kubelet/pods/bd984a03-5cc4-4b56-b490-4e102ed26f9a/volumes"
Jul 07 01:08:38.298627 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.298599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:38.307495 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.307467    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"
Jul 07 01:08:38.361211 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.361078    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48efc560-6e55-4eac-a5b1-0999934fdc85-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:08:38.361211 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.361115    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbh2p\" (UniqueName: \"kubernetes.io/projected/48efc560-6e55-4eac-a5b1-0999934fdc85-kube-api-access-hbh2p\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:08:38.457769 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.457740    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"]
Jul 07 01:08:38.460879 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:38.460853    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062f00db_6780_4cd1_b6c1_26561deaec56.slice/crio-08d5a17b3a19eb8831b08cc16698245782e2de5c62b9ca74354c88b77b822c7a WatchSource:0}: Error finding container 08d5a17b3a19eb8831b08cc16698245782e2de5c62b9ca74354c88b77b822c7a: Status 404 returned error can't find the container with id 08d5a17b3a19eb8831b08cc16698245782e2de5c62b9ca74354c88b77b822c7a
Jul 07 01:08:38.472276 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:38.472250    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"]
Jul 07 01:08:38.474831 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:38.474807    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9ca5e9_8216_464e_83f5_50cdd6c580d4.slice/crio-a648bdb92891fce318e813b01ac97922f58e388a2aa20aa2ffc726ad84f35aa0 WatchSource:0}: Error finding container a648bdb92891fce318e813b01ac97922f58e388a2aa20aa2ffc726ad84f35aa0: Status 404 returned error can't find the container with id a648bdb92891fce318e813b01ac97922f58e388a2aa20aa2ffc726ad84f35aa0
Jul 07 01:08:39.124338 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.123950    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-88mwl"
Jul 07 01:08:39.124338 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.123988    2562 scope.go:117] "RemoveContainer" containerID="cb74db967f42a868e44df2dc442a07bc8c27831ab16ca19f33dc150cb740c532"
Jul 07 01:08:39.127001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.126969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc" event={"ID":"062f00db-6780-4cd1-b6c1-26561deaec56","Type":"ContainerStarted","Data":"acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d"}
Jul 07 01:08:39.127126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.127061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc" event={"ID":"062f00db-6780-4cd1-b6c1-26561deaec56","Type":"ContainerStarted","Data":"08d5a17b3a19eb8831b08cc16698245782e2de5c62b9ca74354c88b77b822c7a"}
Jul 07 01:08:39.127126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.127092    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:39.128568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.128552    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2z6hc"
Jul 07 01:08:39.130577 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.130552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4" event={"ID":"dc9ca5e9-8216-464e-83f5-50cdd6c580d4","Type":"ContainerStarted","Data":"8e7ecd4cfe629137e046a81aaaf700125fbc98fc3482a8f2e7c958b33d86ddc3"}
Jul 07 01:08:39.130673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.130603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4" event={"ID":"dc9ca5e9-8216-464e-83f5-50cdd6c580d4","Type":"ContainerStarted","Data":"a648bdb92891fce318e813b01ac97922f58e388a2aa20aa2ffc726ad84f35aa0"}
Jul 07 01:08:39.130673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.130653    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"
Jul 07 01:08:39.147881 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.147809    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc" podStartSLOduration=2.147792564 podStartE2EDuration="2.147792564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:39.144886569 +0000 UTC m=+447.419877904" watchObservedRunningTime="2026-07-07 01:08:39.147792564 +0000 UTC m=+447.422783887"
Jul 07 01:08:39.162687 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:39.162644    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4" podStartSLOduration=2.162634209 podStartE2EDuration="2.162634209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:39.161085839 +0000 UTC m=+447.436077171" watchObservedRunningTime="2026-07-07 01:08:39.162634209 +0000 UTC m=+447.437625531"
Jul 07 01:08:50.137968 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.137937    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:08:50.138454 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.138173    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8mn4"
Jul 07 01:08:50.477840 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.477810    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"]
Jul 07 01:08:50.597787 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.597757    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"]
Jul 07 01:08:50.597985 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.597899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:50.668114 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.668084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f677g\" (UniqueName: \"kubernetes.io/projected/e91552f9-315f-407a-bd49-a4bb61f5ae8c-kube-api-access-f677g\") pod \"kuadrant-operator-controller-manager-86878c769f-lcnqx\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:50.668266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.668153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e91552f9-315f-407a-bd49-a4bb61f5ae8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lcnqx\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:50.769110 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.769023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e91552f9-315f-407a-bd49-a4bb61f5ae8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lcnqx\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:50.769265 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.769119    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f677g\" (UniqueName: \"kubernetes.io/projected/e91552f9-315f-407a-bd49-a4bb61f5ae8c-kube-api-access-f677g\") pod \"kuadrant-operator-controller-manager-86878c769f-lcnqx\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:50.769454 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.769427    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e91552f9-315f-407a-bd49-a4bb61f5ae8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lcnqx\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:50.781924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.781903    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f677g\" (UniqueName: \"kubernetes.io/projected/e91552f9-315f-407a-bd49-a4bb61f5ae8c-kube-api-access-f677g\") pod \"kuadrant-operator-controller-manager-86878c769f-lcnqx\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:50.910486 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:50.910453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:52.722002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:52.721836    2562 scope.go:117] "RemoveContainer" containerID="61c63de4f53d5878ca5d712c61f61fce700be92f3ec9eb69a59e3e3293e7ede6"
Jul 07 01:08:52.855353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:52.855325    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"]
Jul 07 01:08:52.882306 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:08:52.882274    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91552f9_315f_407a_bd49_a4bb61f5ae8c.slice/crio-4f57af0b2272fbf78da2f41615540c42bc3a640c0dd13d0eec80d0cdb2e62309 WatchSource:0}: Error finding container 4f57af0b2272fbf78da2f41615540c42bc3a640c0dd13d0eec80d0cdb2e62309: Status 404 returned error can't find the container with id 4f57af0b2272fbf78da2f41615540c42bc3a640c0dd13d0eec80d0cdb2e62309
Jul 07 01:08:53.204450 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:53.204415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn" event={"ID":"797adc9d-d61e-45fa-83b3-5e901b21d407","Type":"ContainerStarted","Data":"79fb9786358054a26c45fbe6d576d09b646fc6832e030e44674480bcb5323d4f"}
Jul 07 01:08:53.207554 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:53.207526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx" event={"ID":"e91552f9-315f-407a-bd49-a4bb61f5ae8c","Type":"ContainerStarted","Data":"ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca"}
Jul 07 01:08:53.207709 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:53.207576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx" event={"ID":"e91552f9-315f-407a-bd49-a4bb61f5ae8c","Type":"ContainerStarted","Data":"4f57af0b2272fbf78da2f41615540c42bc3a640c0dd13d0eec80d0cdb2e62309"}
Jul 07 01:08:53.207709 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:53.207613    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:08:53.219873 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:53.219817    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z58vn" podStartSLOduration=1.963639584 podStartE2EDuration="26.219802689s" totalImagesPullingTime="24.256163105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:53.219129251 +0000 UTC m=+461.494120573" watchObservedRunningTime="2026-07-07 01:08:53.219802689 +0000 UTC m=+461.494794011"
Jul 07 01:08:53.235258 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:08:53.235165    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx" podStartSLOduration=3.235151575 podStartE2EDuration="3.235151575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:08:53.234629484 +0000 UTC m=+461.509620809" watchObservedRunningTime="2026-07-07 01:08:53.235151575 +0000 UTC m=+461.510142897"
Jul 07 01:09:04.216171 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.216141    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:09:04.262452 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.262411    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"]
Jul 07 01:09:04.262909 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.262875    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc" podUID="062f00db-6780-4cd1-b6c1-26561deaec56" containerName="manager" containerID="cri-o://acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d" gracePeriod=10
Jul 07 01:09:04.530616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.530590    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:09:04.674895 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.674856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062f00db-6780-4cd1-b6c1-26561deaec56-extensions-socket-volume\") pod \"062f00db-6780-4cd1-b6c1-26561deaec56\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") "
Jul 07 01:09:04.675058 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.674927    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6452d\" (UniqueName: \"kubernetes.io/projected/062f00db-6780-4cd1-b6c1-26561deaec56-kube-api-access-6452d\") pod \"062f00db-6780-4cd1-b6c1-26561deaec56\" (UID: \"062f00db-6780-4cd1-b6c1-26561deaec56\") "
Jul 07 01:09:04.675254 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.675231    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062f00db-6780-4cd1-b6c1-26561deaec56-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "062f00db-6780-4cd1-b6c1-26561deaec56" (UID: "062f00db-6780-4cd1-b6c1-26561deaec56"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:09:04.677103 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.677082    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062f00db-6780-4cd1-b6c1-26561deaec56-kube-api-access-6452d" (OuterVolumeSpecName: "kube-api-access-6452d") pod "062f00db-6780-4cd1-b6c1-26561deaec56" (UID: "062f00db-6780-4cd1-b6c1-26561deaec56"). InnerVolumeSpecName "kube-api-access-6452d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:09:04.775514 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.775432    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6452d\" (UniqueName: \"kubernetes.io/projected/062f00db-6780-4cd1-b6c1-26561deaec56-kube-api-access-6452d\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:09:04.775514 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:04.775462    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/062f00db-6780-4cd1-b6c1-26561deaec56-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:09:05.270436 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.270392    2562 generic.go:356] "Generic (PLEG): container finished" podID="062f00db-6780-4cd1-b6c1-26561deaec56" containerID="acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d" exitCode=0
Jul 07 01:09:05.270903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.270452    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"
Jul 07 01:09:05.270903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.270490    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc" event={"ID":"062f00db-6780-4cd1-b6c1-26561deaec56","Type":"ContainerDied","Data":"acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d"}
Jul 07 01:09:05.270903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.270546    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc" event={"ID":"062f00db-6780-4cd1-b6c1-26561deaec56","Type":"ContainerDied","Data":"08d5a17b3a19eb8831b08cc16698245782e2de5c62b9ca74354c88b77b822c7a"}
Jul 07 01:09:05.270903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.270570    2562 scope.go:117] "RemoveContainer" containerID="acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d"
Jul 07 01:09:05.280353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.280332    2562 scope.go:117] "RemoveContainer" containerID="acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d"
Jul 07 01:09:05.280725 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:09:05.280673    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d\": container with ID starting with acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d not found: ID does not exist" containerID="acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d"
Jul 07 01:09:05.280810 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.280721    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d"} err="failed to get container status \"acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d\": rpc error: code = NotFound desc = could not find container \"acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d\": container with ID starting with acd7f9e723741527c9d9e36f96450c7ce1770a74dfac0d572ee0ff9abb4b293d not found: ID does not exist"
Jul 07 01:09:05.292194 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.292159    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"]
Jul 07 01:09:05.316710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:05.300503    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-c29jc"]
Jul 07 01:09:06.288755 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:06.288717    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="062f00db-6780-4cd1-b6c1-26561deaec56" path="/var/lib/kubelet/pods/062f00db-6780-4cd1-b6c1-26561deaec56/volumes"
Jul 07 01:09:06.633512 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:06.633438    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:06.822792 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:06.822754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:10.620907 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:10.620878    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:10.676332 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:10.676310    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:22.753080 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:22.753043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:22.852003 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:22.851942    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:22.952372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:22.952338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:23.137270 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.137188    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:09:23.137581 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.137563    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="062f00db-6780-4cd1-b6c1-26561deaec56" containerName="manager"
Jul 07 01:09:23.137714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.137585    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="062f00db-6780-4cd1-b6c1-26561deaec56" containerName="manager"
Jul 07 01:09:23.137714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.137674    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="062f00db-6780-4cd1-b6c1-26561deaec56" containerName="manager"
Jul 07 01:09:23.170799 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.170774    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:09:23.170952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.170832    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:09:23.170952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.170864    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.173506 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.173487    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 01:09:23.324753 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.324727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbxr\" (UniqueName: \"kubernetes.io/projected/d75a8c5f-610b-4fbe-bc02-4221167a0eb9-kube-api-access-stbxr\") pod \"limitador-limitador-54d6d988d5-2xg84\" (UID: \"d75a8c5f-610b-4fbe-bc02-4221167a0eb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.324857 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.324797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a8c5f-610b-4fbe-bc02-4221167a0eb9-config-file\") pod \"limitador-limitador-54d6d988d5-2xg84\" (UID: \"d75a8c5f-610b-4fbe-bc02-4221167a0eb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.425082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.425023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbxr\" (UniqueName: \"kubernetes.io/projected/d75a8c5f-610b-4fbe-bc02-4221167a0eb9-kube-api-access-stbxr\") pod \"limitador-limitador-54d6d988d5-2xg84\" (UID: \"d75a8c5f-610b-4fbe-bc02-4221167a0eb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.425082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.425072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a8c5f-610b-4fbe-bc02-4221167a0eb9-config-file\") pod \"limitador-limitador-54d6d988d5-2xg84\" (UID: \"d75a8c5f-610b-4fbe-bc02-4221167a0eb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.425652 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.425635    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d75a8c5f-610b-4fbe-bc02-4221167a0eb9-config-file\") pod \"limitador-limitador-54d6d988d5-2xg84\" (UID: \"d75a8c5f-610b-4fbe-bc02-4221167a0eb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.432933 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.432907    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbxr\" (UniqueName: \"kubernetes.io/projected/d75a8c5f-610b-4fbe-bc02-4221167a0eb9-kube-api-access-stbxr\") pod \"limitador-limitador-54d6d988d5-2xg84\" (UID: \"d75a8c5f-610b-4fbe-bc02-4221167a0eb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.481128 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.481103    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:23.562548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.562516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:23.614445 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.614414    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:09:23.616891 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:09:23.616867    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75a8c5f_610b_4fbe_bc02_4221167a0eb9.slice/crio-1ad53be7b9ddb8741ffe169d2726c7991e9ac9911bd0c21ed1e403a556fc8e27 WatchSource:0}: Error finding container 1ad53be7b9ddb8741ffe169d2726c7991e9ac9911bd0c21ed1e403a556fc8e27: Status 404 returned error can't find the container with id 1ad53be7b9ddb8741ffe169d2726c7991e9ac9911bd0c21ed1e403a556fc8e27
Jul 07 01:09:23.626662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.626636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:23.642736 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.642716    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-jww7g"]
Jul 07 01:09:23.647107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.647091    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jww7g"
Jul 07 01:09:23.649455 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.649437    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-97xpr\""
Jul 07 01:09:23.652280 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.652263    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jww7g"]
Jul 07 01:09:23.826560 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.826537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kndp\" (UniqueName: \"kubernetes.io/projected/4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f-kube-api-access-4kndp\") pod \"authorino-75d556bd77-jww7g\" (UID: \"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f\") " pod="kuadrant-system/authorino-75d556bd77-jww7g"
Jul 07 01:09:23.927522 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.927496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kndp\" (UniqueName: \"kubernetes.io/projected/4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f-kube-api-access-4kndp\") pod \"authorino-75d556bd77-jww7g\" (UID: \"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f\") " pod="kuadrant-system/authorino-75d556bd77-jww7g"
Jul 07 01:09:23.935517 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.935494    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kndp\" (UniqueName: \"kubernetes.io/projected/4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f-kube-api-access-4kndp\") pod \"authorino-75d556bd77-jww7g\" (UID: \"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f\") " pod="kuadrant-system/authorino-75d556bd77-jww7g"
Jul 07 01:09:23.957424 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:23.957398    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jww7g"
Jul 07 01:09:24.082818 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:24.082750    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jww7g"]
Jul 07 01:09:24.086728 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:09:24.086692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4adaef3b_8f8f_4952_af5b_1fa2d42c8b2f.slice/crio-f51776b570fba74258705f68999b9891ad5b85d923fcc9711df94a222d9d9787 WatchSource:0}: Error finding container f51776b570fba74258705f68999b9891ad5b85d923fcc9711df94a222d9d9787: Status 404 returned error can't find the container with id f51776b570fba74258705f68999b9891ad5b85d923fcc9711df94a222d9d9787
Jul 07 01:09:24.355590 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:24.355472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84" event={"ID":"d75a8c5f-610b-4fbe-bc02-4221167a0eb9","Type":"ContainerStarted","Data":"1ad53be7b9ddb8741ffe169d2726c7991e9ac9911bd0c21ed1e403a556fc8e27"}
Jul 07 01:09:24.357082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:24.357051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jww7g" event={"ID":"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f","Type":"ContainerStarted","Data":"f51776b570fba74258705f68999b9891ad5b85d923fcc9711df94a222d9d9787"}
Jul 07 01:09:28.378618 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:28.378507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jww7g" event={"ID":"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f","Type":"ContainerStarted","Data":"08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564"}
Jul 07 01:09:28.380071 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:28.380048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84" event={"ID":"d75a8c5f-610b-4fbe-bc02-4221167a0eb9","Type":"ContainerStarted","Data":"17ee2bcd5026262b3eee4f801e516ce54daea7fa63b6b955d048a734f4a99df5"}
Jul 07 01:09:28.380191 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:28.380145    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:28.393264 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:28.393214    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-jww7g" podStartSLOduration=1.385833285 podStartE2EDuration="5.393197384s" totalImagesPullingTime="4.007364099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:09:28.392134708 +0000 UTC m=+496.667126033" watchObservedRunningTime="2026-07-07 01:09:28.393197384 +0000 UTC m=+496.668188704"
Jul 07 01:09:28.408628 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:28.408573    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84" podStartSLOduration=0.94291302 podStartE2EDuration="5.408555313s" totalImagesPullingTime="4.465642293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:09:28.407285671 +0000 UTC m=+496.682276993" watchObservedRunningTime="2026-07-07 01:09:28.408555313 +0000 UTC m=+496.683546635"
Jul 07 01:09:39.386453 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:39.386409    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2xg84"
Jul 07 01:09:45.465142 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:45.465108    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:46.463001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:46.462909    2562 generic.go:356] "Generic (PLEG): container finished" podID="95b615bf-4983-4ded-a9df-4e1b999314b1" containerID="831a72422eb7239ebaffa3706f595f7eb7ea056ff22dadae778c336c3af6d8c8" exitCode=1
Jul 07 01:09:46.463001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:46.462982    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5" event={"ID":"95b615bf-4983-4ded-a9df-4e1b999314b1","Type":"ContainerDied","Data":"831a72422eb7239ebaffa3706f595f7eb7ea056ff22dadae778c336c3af6d8c8"}
Jul 07 01:09:46.463336 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:46.463323    2562 scope.go:117] "RemoveContainer" containerID="831a72422eb7239ebaffa3706f595f7eb7ea056ff22dadae778c336c3af6d8c8"
Jul 07 01:09:47.469533 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:47.469492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5" event={"ID":"95b615bf-4983-4ded-a9df-4e1b999314b1","Type":"ContainerStarted","Data":"e09cc7a33aee3bd5e67b18e39a1dc04f33e29cd0026ac606c237eabfb85bca8b"}
Jul 07 01:09:47.470038 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:47.469730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:09:56.694064 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:56.694027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:57.457062 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:57.457028    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:09:58.476523 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:58.476485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t9vd5"
Jul 07 01:09:59.213783 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.213743    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jww7g"]
Jul 07 01:09:59.214008 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.213982    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-jww7g" podUID="4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f" containerName="authorino" containerID="cri-o://08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564" gracePeriod=30
Jul 07 01:09:59.503166 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.503119    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jww7g"
Jul 07 01:09:59.531666 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.531630    2562 generic.go:356] "Generic (PLEG): container finished" podID="4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f" containerID="08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564" exitCode=0
Jul 07 01:09:59.531875 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.531677    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jww7g"
Jul 07 01:09:59.531875 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.531720    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jww7g" event={"ID":"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f","Type":"ContainerDied","Data":"08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564"}
Jul 07 01:09:59.531875 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.531773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jww7g" event={"ID":"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f","Type":"ContainerDied","Data":"f51776b570fba74258705f68999b9891ad5b85d923fcc9711df94a222d9d9787"}
Jul 07 01:09:59.531875 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.531800    2562 scope.go:117] "RemoveContainer" containerID="08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564"
Jul 07 01:09:59.540457 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.540438    2562 scope.go:117] "RemoveContainer" containerID="08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564"
Jul 07 01:09:59.540812 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:09:59.540792    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564\": container with ID starting with 08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564 not found: ID does not exist" containerID="08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564"
Jul 07 01:09:59.540862 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.540827    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564"} err="failed to get container status \"08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564\": rpc error: code = NotFound desc = could not find container \"08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564\": container with ID starting with 08ea630d0fbbc8f082bb208b5e5fdaafe3359d8d0be32d3bcf05a107e7a3e564 not found: ID does not exist"
Jul 07 01:09:59.604510 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.604474    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kndp\" (UniqueName: \"kubernetes.io/projected/4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f-kube-api-access-4kndp\") pod \"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f\" (UID: \"4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f\") "
Jul 07 01:09:59.606903 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.606870    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f-kube-api-access-4kndp" (OuterVolumeSpecName: "kube-api-access-4kndp") pod "4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f" (UID: "4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f"). InnerVolumeSpecName "kube-api-access-4kndp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:09:59.705975 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.705937    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kndp\" (UniqueName: \"kubernetes.io/projected/4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f-kube-api-access-4kndp\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:09:59.855049 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.855012    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jww7g"]
Jul 07 01:09:59.868775 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:09:59.868744    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jww7g"]
Jul 07 01:10:00.288943 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:00.288903    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f" path="/var/lib/kubelet/pods/4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f/volumes"
Jul 07 01:10:03.381088 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:03.381056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:03.451210 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:03.451160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:18.553941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:18.553905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:18.855758 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:18.855659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:18.956568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:18.956533    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:22.078499 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:22.078461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:22.177387 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:22.177360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:23.884236 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:23.884203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:53.560489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:53.560454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:53.985749 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:53.985668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:54.239820 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.239737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:54.416237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.416202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:54.504372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.504279    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:54.547261 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.547223    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-kcpjr"]
Jul 07 01:10:54.547638 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.547620    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f" containerName="authorino"
Jul 07 01:10:54.547750 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.547641    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f" containerName="authorino"
Jul 07 01:10:54.547812 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.547763    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4adaef3b-8f8f-4952-af5b-1fa2d42c8b2f" containerName="authorino"
Jul 07 01:10:54.551035 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.551011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:54.553877 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.553853    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8q99v\""
Jul 07 01:10:54.554127 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.554110    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 07 01:10:54.566300 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.566275    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-kcpjr"]
Jul 07 01:10:54.598012 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.597981    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:10:54.657403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.657373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7766b9bb-c719-4447-b7ee-60e0b0d222f2-plugins-config-volume\") pod \"payload-processing-58f86c7798-kcpjr\" (UID: \"7766b9bb-c719-4447-b7ee-60e0b0d222f2\") " pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:54.657577 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.657422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2js6c\" (UniqueName: \"kubernetes.io/projected/7766b9bb-c719-4447-b7ee-60e0b0d222f2-kube-api-access-2js6c\") pod \"payload-processing-58f86c7798-kcpjr\" (UID: \"7766b9bb-c719-4447-b7ee-60e0b0d222f2\") " pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:54.758755 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.758630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7766b9bb-c719-4447-b7ee-60e0b0d222f2-plugins-config-volume\") pod \"payload-processing-58f86c7798-kcpjr\" (UID: \"7766b9bb-c719-4447-b7ee-60e0b0d222f2\") " pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:54.758755 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.758729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2js6c\" (UniqueName: \"kubernetes.io/projected/7766b9bb-c719-4447-b7ee-60e0b0d222f2-kube-api-access-2js6c\") pod \"payload-processing-58f86c7798-kcpjr\" (UID: \"7766b9bb-c719-4447-b7ee-60e0b0d222f2\") " pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:54.759420 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.759402    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7766b9bb-c719-4447-b7ee-60e0b0d222f2-plugins-config-volume\") pod \"payload-processing-58f86c7798-kcpjr\" (UID: \"7766b9bb-c719-4447-b7ee-60e0b0d222f2\") " pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:54.766755 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.766723    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2js6c\" (UniqueName: \"kubernetes.io/projected/7766b9bb-c719-4447-b7ee-60e0b0d222f2-kube-api-access-2js6c\") pod \"payload-processing-58f86c7798-kcpjr\" (UID: \"7766b9bb-c719-4447-b7ee-60e0b0d222f2\") " pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:54.862238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:54.862204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:55.005012 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:55.004987    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-kcpjr"]
Jul 07 01:10:55.007757 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:10:55.007719    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7766b9bb_c719_4447_b7ee_60e0b0d222f2.slice/crio-21991568cc36174d62b6404a7470816250f3aff4a771b165387291bfd8970392 WatchSource:0}: Error finding container 21991568cc36174d62b6404a7470816250f3aff4a771b165387291bfd8970392: Status 404 returned error can't find the container with id 21991568cc36174d62b6404a7470816250f3aff4a771b165387291bfd8970392
Jul 07 01:10:55.790474 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:55.790427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-kcpjr" event={"ID":"7766b9bb-c719-4447-b7ee-60e0b0d222f2","Type":"ContainerStarted","Data":"21991568cc36174d62b6404a7470816250f3aff4a771b165387291bfd8970392"}
Jul 07 01:10:56.797228 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:56.797141    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-kcpjr" event={"ID":"7766b9bb-c719-4447-b7ee-60e0b0d222f2","Type":"ContainerStarted","Data":"3125604df20b83ae34d938872dec84cf3089383fda34b966f7800e49bd6be77d"}
Jul 07 01:10:56.797598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:56.797271    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:10:56.813292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:56.813233    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-kcpjr" podStartSLOduration=1.29125165 podStartE2EDuration="2.813215366s" totalImagesPullingTime="1.521963716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:10:56.812208877 +0000 UTC m=+585.087200201" watchObservedRunningTime="2026-07-07 01:10:56.813215366 +0000 UTC m=+585.088206688"
Jul 07 01:10:57.655048 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:57.655012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:10:57.755934 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:10:57.755901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:11:05.051627 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:05.051590    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:11:05.146558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:05.146528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:11:07.804523 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:07.804494    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-kcpjr"
Jul 07 01:11:12.171211 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:12.171172    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:11:12.178828 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:12.178803    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:11:15.680466 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.680424    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"]
Jul 07 01:11:15.684674 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.684641    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.687198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.687123    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 01:11:15.688435 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.688416    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ks79w\""
Jul 07 01:11:15.688561 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.688431    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 01:11:15.688561 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.688451    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 07 01:11:15.693621 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.693591    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"]
Jul 07 01:11:15.711945 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.711905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:11:15.834492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.834449    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgs2\" (UniqueName: \"kubernetes.io/projected/5bde9223-820c-4bbe-93de-ae0924d34177-kube-api-access-gkgs2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.834492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.834500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.834770 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.834578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.834770 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.834648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.834770 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.834731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bde9223-820c-4bbe-93de-ae0924d34177-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.834770 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.834751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936170 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936170 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936170 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936479 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bde9223-820c-4bbe-93de-ae0924d34177-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936479 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936479 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936236    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgs2\" (UniqueName: \"kubernetes.io/projected/5bde9223-820c-4bbe-93de-ae0924d34177-kube-api-access-gkgs2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936628    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936777 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936746    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.936835 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.936778    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.938737 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.938702    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bde9223-820c-4bbe-93de-ae0924d34177-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.939155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.939132    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bde9223-820c-4bbe-93de-ae0924d34177-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.944305 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.944276    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgs2\" (UniqueName: \"kubernetes.io/projected/5bde9223-820c-4bbe-93de-ae0924d34177-kube-api-access-gkgs2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62\" (UID: \"5bde9223-820c-4bbe-93de-ae0924d34177\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:15.996861 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:15.996824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:16.150824 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:16.150795    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"]
Jul 07 01:11:16.152979 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:11:16.152948    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde9223_820c_4bbe_93de_ae0924d34177.slice/crio-32b49b6e5e1ad2d4589edb11bac8b318743c942ab01d5de85b5bdc3040f9beff WatchSource:0}: Error finding container 32b49b6e5e1ad2d4589edb11bac8b318743c942ab01d5de85b5bdc3040f9beff: Status 404 returned error can't find the container with id 32b49b6e5e1ad2d4589edb11bac8b318743c942ab01d5de85b5bdc3040f9beff
Jul 07 01:11:16.154567 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:16.154546    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 01:11:16.228206 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:16.228163    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:11:16.884461 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:16.884429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62" event={"ID":"5bde9223-820c-4bbe-93de-ae0924d34177","Type":"ContainerStarted","Data":"32b49b6e5e1ad2d4589edb11bac8b318743c942ab01d5de85b5bdc3040f9beff"}
Jul 07 01:11:17.890290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:17.890252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62" event={"ID":"5bde9223-820c-4bbe-93de-ae0924d34177","Type":"ContainerStarted","Data":"212453b86043c88eeb85b09f9b378c6fe517e86db7c0374fd5b149c9d41d7590"}
Jul 07 01:11:17.906883 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:17.906826    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62" podStartSLOduration=1.7164682610000002 podStartE2EDuration="2.906810805s" totalImagesPullingTime="1.190342544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:11:17.906067264 +0000 UTC m=+606.181058587" watchObservedRunningTime="2026-07-07 01:11:17.906810805 +0000 UTC m=+606.181802127"
Jul 07 01:11:25.997365 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:25.997304    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:25.997913 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:25.997426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:26.009829 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:26.009801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:26.943769 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:26.943738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62"
Jul 07 01:11:28.303247 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:28.303206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:11:29.160661 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:29.160623    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:11:32.937595 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:32.937551    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:11:52.603871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:52.603776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:11:53.772694 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:11:53.772655    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:12:04.876578 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.876541    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"]
Jul 07 01:12:04.879811 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.879788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:04.882316 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.882291    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 07 01:12:04.887964 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.887754    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"]
Jul 07 01:12:04.905500 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.905457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:12:04.941763 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.941729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:04.941970 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.941789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:04.941970 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.941912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:04.941970 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.941955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmktp\" (UniqueName: \"kubernetes.io/projected/a6c09bc1-dfe4-436a-9eb5-40766354c96b-kube-api-access-tmktp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:04.942103 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.942067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:04.942103 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:04.942095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c09bc1-dfe4-436a-9eb5-40766354c96b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.042921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.042880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.042921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.042920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c09bc1-dfe4-436a-9eb5-40766354c96b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.043163 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.042944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.043163 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.042986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.043163 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.043023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.043163 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.043048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmktp\" (UniqueName: \"kubernetes.io/projected/a6c09bc1-dfe4-436a-9eb5-40766354c96b-kube-api-access-tmktp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.043427 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.043398    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.043534 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.043455    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.043534 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.043519    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.045337 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.045306    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c09bc1-dfe4-436a-9eb5-40766354c96b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.045575 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.045559    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c09bc1-dfe4-436a-9eb5-40766354c96b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.051027 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.051001    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmktp\" (UniqueName: \"kubernetes.io/projected/a6c09bc1-dfe4-436a-9eb5-40766354c96b-kube-api-access-tmktp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn\" (UID: \"a6c09bc1-dfe4-436a-9eb5-40766354c96b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.192518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.192421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:05.328324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:05.328281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"]
Jul 07 01:12:05.332108 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:12:05.332079    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c09bc1_dfe4_436a_9eb5_40766354c96b.slice/crio-c8d063b5d33b1d28b062096b86d08fbd62b0b15d07b376d0c29dd2fe4e638158 WatchSource:0}: Error finding container c8d063b5d33b1d28b062096b86d08fbd62b0b15d07b376d0c29dd2fe4e638158: Status 404 returned error can't find the container with id c8d063b5d33b1d28b062096b86d08fbd62b0b15d07b376d0c29dd2fe4e638158
Jul 07 01:12:06.115280 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:06.115245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn" event={"ID":"a6c09bc1-dfe4-436a-9eb5-40766354c96b","Type":"ContainerStarted","Data":"f134748581af294b0a3bffaf882911ba0865a2bd1a63424bf5b3b0438b192528"}
Jul 07 01:12:06.115734 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:06.115297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn" event={"ID":"a6c09bc1-dfe4-436a-9eb5-40766354c96b","Type":"ContainerStarted","Data":"c8d063b5d33b1d28b062096b86d08fbd62b0b15d07b376d0c29dd2fe4e638158"}
Jul 07 01:12:06.132567 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:06.132515    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn" podStartSLOduration=1.901319129 podStartE2EDuration="2.132495643s" totalImagesPullingTime="231.176514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:12:06.131084371 +0000 UTC m=+654.406075729" watchObservedRunningTime="2026-07-07 01:12:06.132495643 +0000 UTC m=+654.407486964"
Jul 07 01:12:06.528998 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:06.528959    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:12:15.192824 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:15.192764    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:15.193438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:15.192855    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:15.206005 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:15.205974    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:16.172629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:16.172592    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn"
Jul 07 01:12:24.068631 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:24.068589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:12:24.166158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:24.166121    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:12:27.090879 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:27.090844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:12:29.164236 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:29.164205    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:12:34.375356 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.375324    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"]
Jul 07 01:12:34.379174 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.379150    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.381807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.381783    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 01:12:34.390275 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.390247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"]
Jul 07 01:12:34.405628 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.405599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:12:34.498139 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.498103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwngq\" (UniqueName: \"kubernetes.io/projected/53054bed-a718-4372-a0fa-218fc85ace26-kube-api-access-hwngq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.498139 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.498141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.498349 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.498172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.498349 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.498236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.498418 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.498354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.498418 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.498374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53054bed-a718-4372-a0fa-218fc85ace26-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599029 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.598986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwngq\" (UniqueName: \"kubernetes.io/projected/53054bed-a718-4372-a0fa-218fc85ace26-kube-api-access-hwngq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599029 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599341 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599341 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599341 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599341 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53054bed-a718-4372-a0fa-218fc85ace26-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599534 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599519    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599664 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599642    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.599755 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.599665    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.601520 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.601500    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53054bed-a718-4372-a0fa-218fc85ace26-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.601851 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.601834    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53054bed-a718-4372-a0fa-218fc85ace26-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.606758 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.606732    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwngq\" (UniqueName: \"kubernetes.io/projected/53054bed-a718-4372-a0fa-218fc85ace26-kube-api-access-hwngq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l\" (UID: \"53054bed-a718-4372-a0fa-218fc85ace26\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.688292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.688191    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:34.834569 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:34.834535    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"]
Jul 07 01:12:34.837146 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:12:34.837107    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53054bed_a718_4372_a0fa_218fc85ace26.slice/crio-7cded0e589e9f63a8ee8181852d4b79c651219faa15bf0bbd27d273548802c27 WatchSource:0}: Error finding container 7cded0e589e9f63a8ee8181852d4b79c651219faa15bf0bbd27d273548802c27: Status 404 returned error can't find the container with id 7cded0e589e9f63a8ee8181852d4b79c651219faa15bf0bbd27d273548802c27
Jul 07 01:12:35.251502 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:35.251470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l" event={"ID":"53054bed-a718-4372-a0fa-218fc85ace26","Type":"ContainerStarted","Data":"d889b1be40372a92e7205bf1d54ba4fa99dad0ec541ac1ca2bfa6cb5a3faacc3"}
Jul 07 01:12:35.251715 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:35.251511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l" event={"ID":"53054bed-a718-4372-a0fa-218fc85ace26","Type":"ContainerStarted","Data":"7cded0e589e9f63a8ee8181852d4b79c651219faa15bf0bbd27d273548802c27"}
Jul 07 01:12:35.268833 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:35.268781    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l" podStartSLOduration=1.058597278 podStartE2EDuration="1.268760728s" totalImagesPullingTime="210.16345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:12:35.268572471 +0000 UTC m=+683.543563795" watchObservedRunningTime="2026-07-07 01:12:35.268760728 +0000 UTC m=+683.543752050"
Jul 07 01:12:37.131121 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:37.131083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:12:38.273696 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:38.273642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:12:44.688660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:44.688626    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:44.688660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:44.688671    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:44.701227 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:44.701198    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:45.307527 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:45.307493    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l"
Jul 07 01:12:53.545934 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:12:53.545900    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:14:45.346632 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:14:45.346555    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:16:12.198764 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:16:12.198660    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:16:12.204222 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:16:12.204197    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:16:27.653138 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:16:27.653106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:16:28.937870 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:16:28.937823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:16:56.275410 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:16:56.275369    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:17:05.671415 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:05.671376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:17:17.470191 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:17.470148    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:17:26.375823 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:26.375781    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:17:28.821521 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:28.821479    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:17:28.918953 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:28.918914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:17:37.272043 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:37.271956    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:17:39.119458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:39.119415    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:17:41.101435 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:17:41.101396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:05.775886 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:05.775840    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:18:17.167419 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:17.167386    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:18:21.888313 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:21.888260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:21.986472 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:21.986427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:22.096958 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:22.096919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:22.185895 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:22.185803    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:27.671051 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:27.671011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:27.773250 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:27.773216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:27.870915 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:27.870885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:28.087941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:28.087901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:33.579164 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:33.579120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:33.679785 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:33.679752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:18:50.870023 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:50.869984    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:18:59.874450 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:18:59.874409    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:19:08.870373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:19:08.870287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:19:17.876051 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:19:17.876010    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:19:28.368413 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:19:28.368376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:19:37.572895 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:19:37.572851    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:19:48.973475 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:19:48.973435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:19:56.471871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:19:56.471835    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:20:04.182198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:20:04.182153    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:20:12.070423 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:20:12.070378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:20:27.723795 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:20:27.723752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:20:36.566708 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:20:36.566598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:20:40.871337 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:20:40.871301    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:20:42.972184 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:20:42.972137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:21:00.268092 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:00.268053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:21:10.471277 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:10.471231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:21:12.235610 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:12.235578    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:21:12.239482 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:12.239459    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:21:26.171130 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:26.171087    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:21:34.983548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:34.983514    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:21:46.477218 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:46.477176    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:21:56.074860 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:21:56.074823    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:22:05.269946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:05.269866    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:22:30.203746 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:30.203709    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"]
Jul 07 01:22:30.204334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:30.203918    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx" podUID="e91552f9-315f-407a-bd49-a4bb61f5ae8c" containerName="manager" containerID="cri-o://ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca" gracePeriod=10
Jul 07 01:22:32.871168 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:32.871140    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:22:32.952432 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:32.952398    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f677g\" (UniqueName: \"kubernetes.io/projected/e91552f9-315f-407a-bd49-a4bb61f5ae8c-kube-api-access-f677g\") pod \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") "
Jul 07 01:22:32.952625 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:32.952497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e91552f9-315f-407a-bd49-a4bb61f5ae8c-extensions-socket-volume\") pod \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\" (UID: \"e91552f9-315f-407a-bd49-a4bb61f5ae8c\") "
Jul 07 01:22:32.952863 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:32.952839    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91552f9-315f-407a-bd49-a4bb61f5ae8c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e91552f9-315f-407a-bd49-a4bb61f5ae8c" (UID: "e91552f9-315f-407a-bd49-a4bb61f5ae8c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:22:32.954783 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:32.954752    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91552f9-315f-407a-bd49-a4bb61f5ae8c-kube-api-access-f677g" (OuterVolumeSpecName: "kube-api-access-f677g") pod "e91552f9-315f-407a-bd49-a4bb61f5ae8c" (UID: "e91552f9-315f-407a-bd49-a4bb61f5ae8c"). InnerVolumeSpecName "kube-api-access-f677g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:22:33.029069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.028972    2562 generic.go:356] "Generic (PLEG): container finished" podID="e91552f9-315f-407a-bd49-a4bb61f5ae8c" containerID="ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca" exitCode=0
Jul 07 01:22:33.029069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.029021    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"
Jul 07 01:22:33.029069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.029041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx" event={"ID":"e91552f9-315f-407a-bd49-a4bb61f5ae8c","Type":"ContainerDied","Data":"ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca"}
Jul 07 01:22:33.029305 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.029086    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx" event={"ID":"e91552f9-315f-407a-bd49-a4bb61f5ae8c","Type":"ContainerDied","Data":"4f57af0b2272fbf78da2f41615540c42bc3a640c0dd13d0eec80d0cdb2e62309"}
Jul 07 01:22:33.029305 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.029108    2562 scope.go:117] "RemoveContainer" containerID="ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca"
Jul 07 01:22:33.038537 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.038507    2562 scope.go:117] "RemoveContainer" containerID="ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca"
Jul 07 01:22:33.038876 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:22:33.038859    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca\": container with ID starting with ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca not found: ID does not exist" containerID="ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca"
Jul 07 01:22:33.038928 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.038885    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca"} err="failed to get container status \"ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca\": rpc error: code = NotFound desc = could not find container \"ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca\": container with ID starting with ab10fe893c950f2a2dc8672c833e3cf53c207762a806c08622ed8bb96c0093ca not found: ID does not exist"
Jul 07 01:22:33.049723 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.049669    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"]
Jul 07 01:22:33.053137 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.053113    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lcnqx"]
Jul 07 01:22:33.053454 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.053432    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e91552f9-315f-407a-bd49-a4bb61f5ae8c-extensions-socket-volume\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:22:33.053519 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:33.053461    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f677g\" (UniqueName: \"kubernetes.io/projected/e91552f9-315f-407a-bd49-a4bb61f5ae8c-kube-api-access-f677g\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:22:34.289547 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:22:34.289509    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e91552f9-315f-407a-bd49-a4bb61f5ae8c" path="/var/lib/kubelet/pods/e91552f9-315f-407a-bd49-a4bb61f5ae8c/volumes"
Jul 07 01:23:02.646384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:23:02.646349    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:23:04.049546 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:23:04.049509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:23:41.507516 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:23:41.507407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:23:53.048598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:23:53.048557    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:24:04.540438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:24:04.540403    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:24:12.838928 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:24:12.838890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:24:23.537369 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:24:23.537335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:24:34.642384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:24:34.642344    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:24:43.935779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:24:43.935736    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:24:52.040975 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:24:52.040939    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:25:01.040524 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:25:01.040477    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:25:06.995049 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:25:06.994965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:25:08.647069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:25:08.647032    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:25:17.776899 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:25:17.776855    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:25:19.443882 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:25:19.443728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:25:27.037719 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:25:27.037668    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:25:38.739036 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:25:38.738997    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:26:01.244954 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:01.244912    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:26:10.446929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:10.446886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:26:12.263955 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:12.263929    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:26:12.264403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:12.264240    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:26:20.236022 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:20.235981    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:26:31.031819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:31.031780    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:26:46.740062 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:46.740025    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:26:54.555787 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:54.555745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:26:54.642369 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:54.642332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:26:56.642719 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:26:56.642667    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:27:09.741383 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:27:09.741337    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:27:20.553012 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:27:20.552972    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:27:29.640296 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:27:29.640255    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:27:39.038560 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:27:39.038522    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:27:52.139755 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:27:52.139717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:28:01.937217 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:01.937170    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:28:08.187336 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:08.187254    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:28:09.329178 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:09.329142    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:28:11.437662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:11.437619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:28:23.544336 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:23.544300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:28:32.738322 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:32.738279    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:28:39.541267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:39.541232    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:28:54.042289 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:28:54.042250    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:29:06.543592 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:06.543548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:29:19.638113 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:19.638073    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:29:22.119886 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:22.119844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:27.405775 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:27.405742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:32.039119 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:32.039087    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:29:35.652745 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:35.652628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:40.725887 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:40.725852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:40.740400 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:40.740363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:29:43.463070 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:43.463031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:43.563126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:43.563092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:43.891536 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:43.891498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:49.526208 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.526145    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:49.581546 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.581508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:49.685197 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.685163    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"]
Jul 07 01:29:49.685558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.685538    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91552f9-315f-407a-bd49-a4bb61f5ae8c" containerName="manager"
Jul 07 01:29:49.685558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.685555    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91552f9-315f-407a-bd49-a4bb61f5ae8c" containerName="manager"
Jul 07 01:29:49.685745 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.685610    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e91552f9-315f-407a-bd49-a4bb61f5ae8c" containerName="manager"
Jul 07 01:29:49.688645 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.688624    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.691304 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.691285    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:29:49.691419 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.691365    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-afa7395d-openshift-default-dockercfg-vkccm\""
Jul 07 01:29:49.709504 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.709472    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:49.710304 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.710284    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"]
Jul 07 01:29:49.763713 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-socket\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.763713 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istiod-ca-cert\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.763960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-token\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.763960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-podinfo\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.763960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtq5w\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-kube-api-access-vtq5w\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.763960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-data\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.763960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-envoy\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.764133 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-credential-socket\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.764133 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.763993    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-certs\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.804043 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.803955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:49.865204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-socket\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istiod-ca-cert\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-token\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-podinfo\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtq5w\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-kube-api-access-vtq5w\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-data\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865369    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-envoy\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-credential-socket\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-certs\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865645    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-socket\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.865951 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865923    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-data\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.866086 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.865953    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-certs\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.866086 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.866017    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-credential-socket\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.866086 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.866025    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istiod-ca-cert\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.867916 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.867895    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-envoy\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.868127 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.868110    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-podinfo\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.873522 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.873489    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-token\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:49.873624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:49.873554    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtq5w\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-kube-api-access-vtq5w\") pod \"e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") " pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:50.001305 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:50.001268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:50.152449 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:50.152420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"]
Jul 07 01:29:50.154432 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:29:50.154402    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac2ff1c_7d10_4c84_b001_e620c35032ad.slice/crio-c7b228468c891fa7a069e291bae0699e208db042a20b6a74853404c0eba53709 WatchSource:0}: Error finding container c7b228468c891fa7a069e291bae0699e208db042a20b6a74853404c0eba53709: Status 404 returned error can't find the container with id c7b228468c891fa7a069e291bae0699e208db042a20b6a74853404c0eba53709
Jul 07 01:29:50.156612 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:50.156595    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 01:29:50.811273 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:50.811227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:50.886742 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:50.886672    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:50.956639 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:50.956602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:51.026367 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:51.026312    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq" event={"ID":"8ac2ff1c-7d10-4c84-b001-e620c35032ad","Type":"ContainerStarted","Data":"c7b228468c891fa7a069e291bae0699e208db042a20b6a74853404c0eba53709"}
Jul 07 01:29:51.056579 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:51.056533    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:51.345329 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:51.345292    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:29:51.370129 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:51.370100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:51.471092 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:51.471061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:52.744465 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:52.744429    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:29:52.744780 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:52.744498    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:29:52.744780 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:52.744527    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:29:53.038751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:53.038627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq" event={"ID":"8ac2ff1c-7d10-4c84-b001-e620c35032ad","Type":"ContainerStarted","Data":"75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040"}
Jul 07 01:29:53.058267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:53.058216    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq" podStartSLOduration=1.470847042 podStartE2EDuration="4.058194158s" totalImagesPullingTime="2.587347116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:29:53.057420851 +0000 UTC m=+1721.332412173" watchObservedRunningTime="2026-07-07 01:29:53.058194158 +0000 UTC m=+1721.333185482"
Jul 07 01:29:53.188342 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:53.188295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:54.002437 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:54.002397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:54.007458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:54.007429    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:54.043023 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:54.042987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:54.044180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:54.044161    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:29:55.895993 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:55.895955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:56.048825 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:56.048786    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"]
Jul 07 01:29:56.052180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:56.052147    2562 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq" secret="" err="secret \"e2e-ait-afa7395d-openshift-default-dockercfg-vkccm\" not found"
Jul 07 01:29:56.069179 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:56.069141    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:56.170127 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:56.170035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:56.292004 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:56.291975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:29:57.057267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:57.057209    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq" podUID="8ac2ff1c-7d10-4c84-b001-e620c35032ad" containerName="istio-proxy" containerID="cri-o://75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040" gracePeriod=30
Jul 07 01:29:57.574209 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:29:57.574179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:00.131521 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.131485    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723130-8chtv"]
Jul 07 01:30:00.135070 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.135051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv"
Jul 07 01:30:00.137625 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.137605    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jtxhz\""
Jul 07 01:30:00.143931 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.143894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723130-8chtv"]
Jul 07 01:30:00.263968 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.263927    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsttj\" (UniqueName: \"kubernetes.io/projected/c822a053-9353-459a-914b-97cd0a1ddae0-kube-api-access-dsttj\") pod \"maas-api-key-cleanup-29723130-8chtv\" (UID: \"c822a053-9353-459a-914b-97cd0a1ddae0\") " pod="opendatahub/maas-api-key-cleanup-29723130-8chtv"
Jul 07 01:30:00.365033 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.364994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsttj\" (UniqueName: \"kubernetes.io/projected/c822a053-9353-459a-914b-97cd0a1ddae0-kube-api-access-dsttj\") pod \"maas-api-key-cleanup-29723130-8chtv\" (UID: \"c822a053-9353-459a-914b-97cd0a1ddae0\") " pod="opendatahub/maas-api-key-cleanup-29723130-8chtv"
Jul 07 01:30:00.373488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.373457    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsttj\" (UniqueName: \"kubernetes.io/projected/c822a053-9353-459a-914b-97cd0a1ddae0-kube-api-access-dsttj\") pod \"maas-api-key-cleanup-29723130-8chtv\" (UID: \"c822a053-9353-459a-914b-97cd0a1ddae0\") " pod="opendatahub/maas-api-key-cleanup-29723130-8chtv"
Jul 07 01:30:00.446184 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.446088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv"
Jul 07 01:30:00.586084 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:00.585963    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723130-8chtv"]
Jul 07 01:30:00.588392 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:30:00.588366    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc822a053_9353_459a_914b_97cd0a1ddae0.slice/crio-bab819ad9f4c5ea0c7fc3d09c93bf6b46b736602ababf471a2d35a91bfd275ad WatchSource:0}: Error finding container bab819ad9f4c5ea0c7fc3d09c93bf6b46b736602ababf471a2d35a91bfd275ad: Status 404 returned error can't find the container with id bab819ad9f4c5ea0c7fc3d09c93bf6b46b736602ababf471a2d35a91bfd275ad
Jul 07 01:30:01.077867 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:01.077835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv" event={"ID":"c822a053-9353-459a-914b-97cd0a1ddae0","Type":"ContainerStarted","Data":"bab819ad9f4c5ea0c7fc3d09c93bf6b46b736602ababf471a2d35a91bfd275ad"}
Jul 07 01:30:01.782425 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:01.782392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:01.942470 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:01.942394    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"]
Jul 07 01:30:01.946542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:01.946513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:01.949181 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:01.949155    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-30ad23ec-openshift-default-dockercfg-dgtf4\""
Jul 07 01:30:01.959751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:01.959649    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"]
Jul 07 01:30:02.041453 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.041363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:02.080125 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-envoy\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-podinfo\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-credential-socket\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8c9\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-kube-api-access-vc8c9\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080435 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-certs\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080435 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-socket\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080511 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-data\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080511 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istiod-ca-cert\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.080578 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.080515    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-token\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.141287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.141251    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:02.180960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.180923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-token\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181144 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.180981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-envoy\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181144 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-podinfo\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-credential-socket\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8c9\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-kube-api-access-vc8c9\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-certs\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181439 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-socket\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181439 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-data\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181549 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istiod-ca-cert\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.181938 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.181865    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-credential-socket\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.182320 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.182271    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-certs\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.182559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.182535    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-socket\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.182658 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.182554    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-data\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.182762 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.182737    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istiod-ca-cert\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.183771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.183744    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-envoy\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.184645 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.184619    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-podinfo\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.189818 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.189759    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-token\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.190363 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.190338    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8c9\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-kube-api-access-vc8c9\") pod \"e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") " pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.263269 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.263231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:02.321490 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.321162    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:30:02.440415 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.440379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"]
Jul 07 01:30:02.444395 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:30:02.444347    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75e9a6b_b7ff_4f37_8742_28258b2d7b26.slice/crio-6c4951bab9ade9c72846e1cf08dff32f67e18f024fe4cf6fe0123927411176ce WatchSource:0}: Error finding container 6c4951bab9ade9c72846e1cf08dff32f67e18f024fe4cf6fe0123927411176ce: Status 404 returned error can't find the container with id 6c4951bab9ade9c72846e1cf08dff32f67e18f024fe4cf6fe0123927411176ce
Jul 07 01:30:02.447149 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.447111    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:30:02.447244 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.447208    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:30:02.447292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.447251    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:30:02.484769 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.484743    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istiod-ca-cert\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.484918 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.484795    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtq5w\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-kube-api-access-vtq5w\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.484918 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.484825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-data\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.485026 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.484960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-token\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.485069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485045    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-podinfo\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.485119 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485078    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:02.485119 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485087    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-certs\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.485221 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-socket\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.485221 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-envoy\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.485312 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485233    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-credential-socket\") pod \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\" (UID: \"8ac2ff1c-7d10-4c84-b001-e620c35032ad\") "
Jul 07 01:30:02.485659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485409    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:30:02.485659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485514    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.485659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485532    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.485659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485578    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:02.485659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485623    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:02.486149 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.485769    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:02.487704 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.487652    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-kube-api-access-vtq5w" (OuterVolumeSpecName: "kube-api-access-vtq5w") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "kube-api-access-vtq5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:30:02.488388 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.488365    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:30:02.488469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.488391    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:30:02.488469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.488398    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ac2ff1c-7d10-4c84-b001-e620c35032ad" (UID: "8ac2ff1c-7d10-4c84-b001-e620c35032ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:02.586238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.586143    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.586238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.586186    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.586238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.586201    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.586238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.586214    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.586238 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.586229    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac2ff1c-7d10-4c84-b001-e620c35032ad-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.586732 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.586246    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtq5w\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-kube-api-access-vtq5w\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:02.586732 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:02.586260    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac2ff1c-7d10-4c84-b001-e620c35032ad-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:03.090662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.090626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:03.091575 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.091538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh" event={"ID":"d75e9a6b-b7ff-4f37-8742-28258b2d7b26","Type":"ContainerStarted","Data":"f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85"}
Jul 07 01:30:03.091733 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.091614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh" event={"ID":"d75e9a6b-b7ff-4f37-8742-28258b2d7b26","Type":"ContainerStarted","Data":"6c4951bab9ade9c72846e1cf08dff32f67e18f024fe4cf6fe0123927411176ce"}
Jul 07 01:30:03.093034 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.093004    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ac2ff1c-7d10-4c84-b001-e620c35032ad" containerID="75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040" exitCode=0
Jul 07 01:30:03.093160 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.093049    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"
Jul 07 01:30:03.093160 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.093087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq" event={"ID":"8ac2ff1c-7d10-4c84-b001-e620c35032ad","Type":"ContainerDied","Data":"75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040"}
Jul 07 01:30:03.093160 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.093151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq" event={"ID":"8ac2ff1c-7d10-4c84-b001-e620c35032ad","Type":"ContainerDied","Data":"c7b228468c891fa7a069e291bae0699e208db042a20b6a74853404c0eba53709"}
Jul 07 01:30:03.093327 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.093183    2562 scope.go:117] "RemoveContainer" containerID="75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040"
Jul 07 01:30:03.117356 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.117298    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh" podStartSLOduration=2.117277494 podStartE2EDuration="2.117277494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:30:03.115411152 +0000 UTC m=+1731.390402474" watchObservedRunningTime="2026-07-07 01:30:03.117277494 +0000 UTC m=+1731.392268799"
Jul 07 01:30:03.136018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.135981    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"]
Jul 07 01:30:03.150854 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.150817    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-afa7395d-openshift-default-67876bc855-bw9sq"]
Jul 07 01:30:03.176025 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.175962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:03.263878 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.263835    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:03.270353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.270319    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:03.288330 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.288270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:03.338500 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.338332    2562 scope.go:117] "RemoveContainer" containerID="75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040"
Jul 07 01:30:03.338772 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:30:03.338745    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040\": container with ID starting with 75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040 not found: ID does not exist" containerID="75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040"
Jul 07 01:30:03.338850 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.338787    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040"} err="failed to get container status \"75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040\": rpc error: code = NotFound desc = could not find container \"75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040\": container with ID starting with 75e016efc0eb33de805ab5ae77f3682e7d4c5939b9a3b6dc016d0717bea2e040 not found: ID does not exist"
Jul 07 01:30:03.749531 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.749493    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:03.849623 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:03.849592    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:04.100722 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:04.100661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv" event={"ID":"c822a053-9353-459a-914b-97cd0a1ddae0","Type":"ContainerStarted","Data":"d20741e783f8eda80bf1dc4904c6b689e58207fcf6c60c2f22d9694f0b19805d"}
Jul 07 01:30:04.101161 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:04.100757    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:04.101788 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:04.101770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:04.115535 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:04.115486    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv" podStartSLOduration=1.307334085 podStartE2EDuration="4.115469545s" totalImagesPullingTime="2.80813546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:30:04.114399849 +0000 UTC m=+1732.389391173" watchObservedRunningTime="2026-07-07 01:30:04.115469545 +0000 UTC m=+1732.390460866"
Jul 07 01:30:04.289932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:04.289887    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac2ff1c-7d10-4c84-b001-e620c35032ad" path="/var/lib/kubelet/pods/8ac2ff1c-7d10-4c84-b001-e620c35032ad/volumes"
Jul 07 01:30:05.107021 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:05.106985    2562 generic.go:356] "Generic (PLEG): container finished" podID="c822a053-9353-459a-914b-97cd0a1ddae0" containerID="d20741e783f8eda80bf1dc4904c6b689e58207fcf6c60c2f22d9694f0b19805d" exitCode=0
Jul 07 01:30:05.107469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:05.107071    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv" event={"ID":"c822a053-9353-459a-914b-97cd0a1ddae0","Type":"ContainerDied","Data":"d20741e783f8eda80bf1dc4904c6b689e58207fcf6c60c2f22d9694f0b19805d"}
Jul 07 01:30:05.165850 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:05.165820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:06.249549 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:06.249519    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv"
Jul 07 01:30:06.420257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:06.420166    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsttj\" (UniqueName: \"kubernetes.io/projected/c822a053-9353-459a-914b-97cd0a1ddae0-kube-api-access-dsttj\") pod \"c822a053-9353-459a-914b-97cd0a1ddae0\" (UID: \"c822a053-9353-459a-914b-97cd0a1ddae0\") "
Jul 07 01:30:06.422491 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:06.422464    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c822a053-9353-459a-914b-97cd0a1ddae0-kube-api-access-dsttj" (OuterVolumeSpecName: "kube-api-access-dsttj") pod "c822a053-9353-459a-914b-97cd0a1ddae0" (UID: "c822a053-9353-459a-914b-97cd0a1ddae0"). InnerVolumeSpecName "kube-api-access-dsttj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:30:06.521121 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:06.521068    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsttj\" (UniqueName: \"kubernetes.io/projected/c822a053-9353-459a-914b-97cd0a1ddae0-kube-api-access-dsttj\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:07.117922 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:07.117885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv" event={"ID":"c822a053-9353-459a-914b-97cd0a1ddae0","Type":"ContainerDied","Data":"bab819ad9f4c5ea0c7fc3d09c93bf6b46b736602ababf471a2d35a91bfd275ad"}
Jul 07 01:30:07.117922 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:07.117916    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723130-8chtv"
Jul 07 01:30:07.118157 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:07.117941    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bab819ad9f4c5ea0c7fc3d09c93bf6b46b736602ababf471a2d35a91bfd275ad"
Jul 07 01:30:07.237768 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:07.237731    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:30:08.145431 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:08.145395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:08.458394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:08.458298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:14.197363 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:14.197317    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"]
Jul 07 01:30:14.197886 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:14.197612    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh" podUID="d75e9a6b-b7ff-4f37-8742-28258b2d7b26" containerName="istio-proxy" containerID="cri-o://f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85" gracePeriod=30
Jul 07 01:30:14.225407 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:14.225250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:14.305932 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:14.305895    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:15.557138 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.557108    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:15.605880 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.605854    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-certs\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.605997 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.605894    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-podinfo\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.605997 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.605918    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-token\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.605997 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.605947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istiod-ca-cert\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.606132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606001    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-data\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.606132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-envoy\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.606132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606061    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-credential-socket\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.606132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606095    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-socket\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.606323 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606140    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc8c9\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-kube-api-access-vc8c9\") pod \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\" (UID: \"d75e9a6b-b7ff-4f37-8742-28258b2d7b26\") "
Jul 07 01:30:15.606323 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606231    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:15.606435 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606364    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:30:15.606504 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606483    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.606563 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606505    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.606563 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606549    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:15.607017 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.606945    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:15.607252 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.607229    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-data" (OuterVolumeSpecName: "istio-data") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:15.608466 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.608440    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:30:15.608921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.608875    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-token" (OuterVolumeSpecName: "istio-token") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:30:15.608921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.608886    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-kube-api-access-vc8c9" (OuterVolumeSpecName: "kube-api-access-vc8c9") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "kube-api-access-vc8c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:30:15.609091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.608962    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d75e9a6b-b7ff-4f37-8742-28258b2d7b26" (UID: "d75e9a6b-b7ff-4f37-8742-28258b2d7b26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:15.706967 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.706922    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.706967 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.706960    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.706967 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.706975    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.707224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.706988    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.707224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.707000    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc8c9\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-kube-api-access-vc8c9\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.707224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.707014    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:15.707224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:15.707054    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75e9a6b-b7ff-4f37-8742-28258b2d7b26-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:16.172972 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.172931    2562 generic.go:356] "Generic (PLEG): container finished" podID="d75e9a6b-b7ff-4f37-8742-28258b2d7b26" containerID="f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85" exitCode=0
Jul 07 01:30:16.173190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.172986    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"
Jul 07 01:30:16.173190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.172991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh" event={"ID":"d75e9a6b-b7ff-4f37-8742-28258b2d7b26","Type":"ContainerDied","Data":"f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85"}
Jul 07 01:30:16.173190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.173042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh" event={"ID":"d75e9a6b-b7ff-4f37-8742-28258b2d7b26","Type":"ContainerDied","Data":"6c4951bab9ade9c72846e1cf08dff32f67e18f024fe4cf6fe0123927411176ce"}
Jul 07 01:30:16.173190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.173071    2562 scope.go:117] "RemoveContainer" containerID="f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85"
Jul 07 01:30:16.182838 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.182819    2562 scope.go:117] "RemoveContainer" containerID="f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85"
Jul 07 01:30:16.183136 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:30:16.183112    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85\": container with ID starting with f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85 not found: ID does not exist" containerID="f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85"
Jul 07 01:30:16.183207 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.183147    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85"} err="failed to get container status \"f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85\": rpc error: code = NotFound desc = could not find container \"f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85\": container with ID starting with f62926a29f811d482b332b036c7ebb59df03819e6d7802152ef9845785e10b85 not found: ID does not exist"
Jul 07 01:30:16.194857 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.194831    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"]
Jul 07 01:30:16.198922 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.198898    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-30ad23ec-openshift-default-57f5f599d5-9hjmh"]
Jul 07 01:30:16.289542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.289509    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d75e9a6b-b7ff-4f37-8742-28258b2d7b26" path="/var/lib/kubelet/pods/d75e9a6b-b7ff-4f37-8742-28258b2d7b26/volumes"
Jul 07 01:30:16.678665 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:16.678630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:17.044655 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:17.044619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:30:20.051020 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.050981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:20.151378 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.151335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:20.228523 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228490    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"]
Jul 07 01:30:20.228848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228835    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac2ff1c-7d10-4c84-b001-e620c35032ad" containerName="istio-proxy"
Jul 07 01:30:20.228941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228850    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac2ff1c-7d10-4c84-b001-e620c35032ad" containerName="istio-proxy"
Jul 07 01:30:20.228941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228864    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75e9a6b-b7ff-4f37-8742-28258b2d7b26" containerName="istio-proxy"
Jul 07 01:30:20.228941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228869    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75e9a6b-b7ff-4f37-8742-28258b2d7b26" containerName="istio-proxy"
Jul 07 01:30:20.228941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228882    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c822a053-9353-459a-914b-97cd0a1ddae0" containerName="cleanup"
Jul 07 01:30:20.228941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228888    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c822a053-9353-459a-914b-97cd0a1ddae0" containerName="cleanup"
Jul 07 01:30:20.228941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228944    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c822a053-9353-459a-914b-97cd0a1ddae0" containerName="cleanup"
Jul 07 01:30:20.229259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228954    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac2ff1c-7d10-4c84-b001-e620c35032ad" containerName="istio-proxy"
Jul 07 01:30:20.229259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.228960    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d75e9a6b-b7ff-4f37-8742-28258b2d7b26" containerName="istio-proxy"
Jul 07 01:30:20.235914 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.235883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.238611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.238583    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5ec2bff8-openshift-default-dockercfg-mf6mg\""
Jul 07 01:30:20.238611 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.238585    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:30:20.245008 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.244894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"]
Jul 07 01:30:20.263568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.263536    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:20.324126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.324035    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:20.343287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7s48\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-kube-api-access-n7s48\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343508 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-envoy\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343567 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343549    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-certs\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343636 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-socket\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343721 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-data\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343770 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0793540-a19d-4e72-b0af-a57ba3f0049e-istiod-ca-cert\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343806 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-credential-socket\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-token\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.343842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.343838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-podinfo\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.444676 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-socket\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.444876 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-data\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.444876 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0793540-a19d-4e72-b0af-a57ba3f0049e-istiod-ca-cert\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.444876 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-credential-socket\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.444876 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-token\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.444876 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-podinfo\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.444876 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7s48\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-kube-api-access-n7s48\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.445167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-envoy\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.445167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.444917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-certs\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.445167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.445108    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-socket\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.445334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.445298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-credential-socket\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.445389 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.445347    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-certs\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.445485 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.445462    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-data\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.445544 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.445510    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0793540-a19d-4e72-b0af-a57ba3f0049e-istiod-ca-cert\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.447261 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.447239    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-envoy\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.447617 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.447592    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-podinfo\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.453767 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.453733    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7s48\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-kube-api-access-n7s48\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.454065 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.454043    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-token\") pod \"e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") " pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.548643 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.548607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:20.696134 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.696037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"]
Jul 07 01:30:20.698926 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:30:20.698889    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0793540_a19d_4e72_b0af_a57ba3f0049e.slice/crio-7b7d3b6fe3862c7ba6b043716aae67e5bd4e60d65e83b00b493d5f14c6839f01 WatchSource:0}: Error finding container 7b7d3b6fe3862c7ba6b043716aae67e5bd4e60d65e83b00b493d5f14c6839f01: Status 404 returned error can't find the container with id 7b7d3b6fe3862c7ba6b043716aae67e5bd4e60d65e83b00b493d5f14c6839f01
Jul 07 01:30:20.701097 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.701057    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:30:20.701180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.701149    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:30:20.701220 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:20.701193    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:30:21.206294 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.206255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5" event={"ID":"c0793540-a19d-4e72-b0af-a57ba3f0049e","Type":"ContainerStarted","Data":"3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f"}
Jul 07 01:30:21.206675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.206316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5" event={"ID":"c0793540-a19d-4e72-b0af-a57ba3f0049e","Type":"ContainerStarted","Data":"7b7d3b6fe3862c7ba6b043716aae67e5bd4e60d65e83b00b493d5f14c6839f01"}
Jul 07 01:30:21.229417 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.229355    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5" podStartSLOduration=1.229339347 podStartE2EDuration="1.229339347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:30:21.227534103 +0000 UTC m=+1749.502525426" watchObservedRunningTime="2026-07-07 01:30:21.229339347 +0000 UTC m=+1749.504330749"
Jul 07 01:30:21.359418 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.359388    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:21.462147 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.462069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:21.549237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.549206    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:21.554071 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.554043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:21.561995 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.561968    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:21.794475 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:21.794436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:22.024914 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:22.024871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:22.122255 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:22.122163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:22.211317 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:22.211283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:22.212309 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:22.212287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:23.508413 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:23.508377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:24.638862 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:24.638822    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:30:25.786518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:25.786480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:25.953487 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:25.953449    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"]
Jul 07 01:30:25.953674 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:25.953633    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5" podUID="c0793540-a19d-4e72-b0af-a57ba3f0049e" containerName="istio-proxy" containerID="cri-o://3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f" gracePeriod=30
Jul 07 01:30:25.977313 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:25.977278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:26.077987 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:26.077891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:26.177591 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:26.177555    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:27.835960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:27.835927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:31.764103 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:31.764055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:31.824389 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:31.824330    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:32.219360 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.219330    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:32.319596 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.319562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:32.418867 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.418829    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:32.721492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.721465    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:32.854619 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854520    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7s48\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-kube-api-access-n7s48\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.854619 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854558    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0793540-a19d-4e72-b0af-a57ba3f0049e-istiod-ca-cert\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.854619 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854594    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-envoy\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854624    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-certs\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-podinfo\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854730    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-token\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-socket\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854857    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-data\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854886    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-credential-socket\") pod \"c0793540-a19d-4e72-b0af-a57ba3f0049e\" (UID: \"c0793540-a19d-4e72-b0af-a57ba3f0049e\") "
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.854985    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:32.855231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.855076    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0793540-a19d-4e72-b0af-a57ba3f0049e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:30:32.855998 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.855251    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0793540-a19d-4e72-b0af-a57ba3f0049e-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.855998 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.855272    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.855998 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.855248    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:32.856156 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.856009    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:32.856212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.856148    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:32.857334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.857297    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-kube-api-access-n7s48" (OuterVolumeSpecName: "kube-api-access-n7s48") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "kube-api-access-n7s48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:30:32.857553 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.857528    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:30:32.857648 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.857625    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:30:32.857793 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.857770    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0793540-a19d-4e72-b0af-a57ba3f0049e" (UID: "c0793540-a19d-4e72-b0af-a57ba3f0049e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:30:32.939211 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.939171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:30:32.956055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.956018    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.956055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.956051    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.956055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.956062    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.956274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.956073    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7s48\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-kube-api-access-n7s48\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.956274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.956088    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.956274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.956099    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:32.956274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:32.956106    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0793540-a19d-4e72-b0af-a57ba3f0049e-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:30:33.288851 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.288813    2562 generic.go:356] "Generic (PLEG): container finished" podID="c0793540-a19d-4e72-b0af-a57ba3f0049e" containerID="3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f" exitCode=0
Jul 07 01:30:33.289037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.288867    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"
Jul 07 01:30:33.289037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.288902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5" event={"ID":"c0793540-a19d-4e72-b0af-a57ba3f0049e","Type":"ContainerDied","Data":"3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f"}
Jul 07 01:30:33.289037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.288964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5" event={"ID":"c0793540-a19d-4e72-b0af-a57ba3f0049e","Type":"ContainerDied","Data":"7b7d3b6fe3862c7ba6b043716aae67e5bd4e60d65e83b00b493d5f14c6839f01"}
Jul 07 01:30:33.289037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.288993    2562 scope.go:117] "RemoveContainer" containerID="3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f"
Jul 07 01:30:33.298942 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.298922    2562 scope.go:117] "RemoveContainer" containerID="3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f"
Jul 07 01:30:33.299225 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:30:33.299207    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f\": container with ID starting with 3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f not found: ID does not exist" containerID="3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f"
Jul 07 01:30:33.299301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.299235    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f"} err="failed to get container status \"3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f\": rpc error: code = NotFound desc = could not find container \"3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f\": container with ID starting with 3979bb5dcc508ca1279ed5f7a1ec0e9dfd47630880c2057feac7702061ffa94f not found: ID does not exist"
Jul 07 01:30:33.329193 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.329156    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"]
Jul 07 01:30:33.340014 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.339981    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5ec2bff8-openshift-default-786bfc64bc-9lmr5"]
Jul 07 01:30:33.790408 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:33.790374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:34.291009 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:34.290975    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0793540-a19d-4e72-b0af-a57ba3f0049e" path="/var/lib/kubelet/pods/c0793540-a19d-4e72-b0af-a57ba3f0049e/volumes"
Jul 07 01:30:39.455783 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:39.455752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:39.517212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:39.517183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:39.589067 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:39.589031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:39.689209 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:39.689178    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:39.789905 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:39.789872    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:30:40.151343 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:30:40.151249    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:06.121910 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:06.121826    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:11.761160 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:11.761125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:11.871367 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:11.871320    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:11.961627 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:11.961596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:12.289965 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:12.289934    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:31:12.291406 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:12.291385    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:31:27.841522 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:27.841486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:33.603624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:33.603556    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:33.661660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:33.661602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:33.732021 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:33.731981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:33.831865 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:33.831830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:44.306887 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:44.306834    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:45.099914 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:45.099882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:50.686132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:50.686095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:50.786303 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:50.786270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:50.886896 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:50.886841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:50.988701 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:50.988652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:51.700029 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:51.699996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:51.801885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:51.801843    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:51.898406 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:51.898359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:53.448635 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:53.448590    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:54.141403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:54.141361    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:31:59.090196 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:59.090163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:59.191320 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:59.191286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:31:59.759457 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:31:59.759426    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:02.142742 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:02.142703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:32:05.433535 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.433474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:05.489995 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.489954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:05.657902 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.657869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:05.929246 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.929216    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"]
Jul 07 01:32:05.929554 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.929543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0793540-a19d-4e72-b0af-a57ba3f0049e" containerName="istio-proxy"
Jul 07 01:32:05.929598 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.929556    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0793540-a19d-4e72-b0af-a57ba3f0049e" containerName="istio-proxy"
Jul 07 01:32:05.929632 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.929623    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0793540-a19d-4e72-b0af-a57ba3f0049e" containerName="istio-proxy"
Jul 07 01:32:05.932597 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.932579    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934220 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-envoy\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934355 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-podinfo\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934355 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-credential-socket\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934355 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-socket\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934355 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39b681-516c-4208-8f9c-f944d2b91ed5-istiod-ca-cert\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934553 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-token\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934553 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6c8\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-kube-api-access-lq6c8\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934553 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934501    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-data\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.934668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.934563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-certs\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:05.935438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.935412    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:32:05.935540 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.935438    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4a5d2769-openshift-default-dockercfg-5762s\""
Jul 07 01:32:05.952853 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.952817    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"]
Jul 07 01:32:05.955334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:05.955302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:06.035115 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-certs\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-envoy\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-podinfo\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-credential-socket\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035282 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-socket\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035505 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39b681-516c-4208-8f9c-f944d2b91ed5-istiod-ca-cert\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035505 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-token\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035505 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6c8\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-kube-api-access-lq6c8\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035653 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-data\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035653 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035553    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-certs\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035653 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035586    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-socket\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035859 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035652    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-credential-socket\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.035921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.035871    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-data\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.036467 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.036443    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39b681-516c-4208-8f9c-f944d2b91ed5-istiod-ca-cert\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.037857 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.037838    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-envoy\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.038211 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.038195    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-podinfo\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.043995 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.043967    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-token\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.044120 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.044071    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6c8\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-kube-api-access-lq6c8\") pod \"e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") " pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.243123 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.243075    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:06.392941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.392908    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"]
Jul 07 01:32:06.396608 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:32:06.396574    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f39b681_516c_4208_8f9c_f944d2b91ed5.slice/crio-19759a5afc7e35d1b79d807a62a89718c9a0109db4f34e3fbeb318873088c609 WatchSource:0}: Error finding container 19759a5afc7e35d1b79d807a62a89718c9a0109db4f34e3fbeb318873088c609: Status 404 returned error can't find the container with id 19759a5afc7e35d1b79d807a62a89718c9a0109db4f34e3fbeb318873088c609
Jul 07 01:32:06.398783 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.398746    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:06.398880 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.398821    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:06.398880 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.398849    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:06.607311 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.607214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:06.717156 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.717118    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" event={"ID":"4f39b681-516c-4208-8f9c-f944d2b91ed5","Type":"ContainerStarted","Data":"b20069e74826c8f9a55a36cbe83f8ed6f2e8620b30e2b7ba865208120ae1c30c"}
Jul 07 01:32:06.717330 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.717184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" event={"ID":"4f39b681-516c-4208-8f9c-f944d2b91ed5","Type":"ContainerStarted","Data":"19759a5afc7e35d1b79d807a62a89718c9a0109db4f34e3fbeb318873088c609"}
Jul 07 01:32:06.736406 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:06.736348    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" podStartSLOduration=1.7363309070000001 podStartE2EDuration="1.736330907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:32:06.734206896 +0000 UTC m=+1855.009198217" watchObservedRunningTime="2026-07-07 01:32:06.736330907 +0000 UTC m=+1855.011322229"
Jul 07 01:32:07.243808 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:07.243770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:07.249265 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:07.249234    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:07.722201 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:07.722166    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:07.723434 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:07.723409    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:08.698047 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.698015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:08.756310 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.756273    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"]
Jul 07 01:32:08.760297 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.760270    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.762609 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.762578    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jtxhz\""
Jul 07 01:32:08.762609 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.762608    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-4a5d2769-serving-cert\""
Jul 07 01:32:08.770592 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.770563    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"]
Jul 07 01:32:08.773835 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.773794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:08.834956 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.834917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:08.855760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.855715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzhcg\" (UniqueName: \"kubernetes.io/projected/9695e059-ba1e-4b23-995c-6cf53f616df5-kube-api-access-vzhcg\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.855946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.855809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695e059-ba1e-4b23-995c-6cf53f616df5-openshift-service-ca\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.855946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.855848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.956400 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.956275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.956400 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.956391    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzhcg\" (UniqueName: \"kubernetes.io/projected/9695e059-ba1e-4b23-995c-6cf53f616df5-kube-api-access-vzhcg\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.956653 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:32:08.956433    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-4a5d2769-serving-cert: secret "maas-api-e2e-mt-4a5d2769-serving-cert" not found
Jul 07 01:32:08.956653 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.956457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695e059-ba1e-4b23-995c-6cf53f616df5-openshift-service-ca\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.956653 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:32:08.956500    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls podName:9695e059-ba1e-4b23-995c-6cf53f616df5 nodeName:}" failed. No retries permitted until 2026-07-07 01:32:09.456484002 +0000 UTC m=+1857.731475302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls") pod "maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp" (UID: "9695e059-ba1e-4b23-995c-6cf53f616df5") : secret "maas-api-e2e-mt-4a5d2769-serving-cert" not found
Jul 07 01:32:08.957159 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.957135    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695e059-ba1e-4b23-995c-6cf53f616df5-openshift-service-ca\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:08.965007 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:08.964984    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzhcg\" (UniqueName: \"kubernetes.io/projected/9695e059-ba1e-4b23-995c-6cf53f616df5-kube-api-access-vzhcg\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:09.110636 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:09.110599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:09.231401 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:09.231324    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:09.334456 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:09.334419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:09.460145 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:09.460102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:09.463265 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:09.463233    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls\") pod \"maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") " pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:09.673252 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:09.673153    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:09.817881 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:09.817852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"]
Jul 07 01:32:09.820288 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:32:09.820255    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9695e059_ba1e_4b23_995c_6cf53f616df5.slice/crio-f1411433fe3db919dab44913f1175e5f8e716ca735078a3f1eb3633f2e4c5e03 WatchSource:0}: Error finding container f1411433fe3db919dab44913f1175e5f8e716ca735078a3f1eb3633f2e4c5e03: Status 404 returned error can't find the container with id f1411433fe3db919dab44913f1175e5f8e716ca735078a3f1eb3633f2e4c5e03
Jul 07 01:32:10.742075 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:10.742038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp" event={"ID":"9695e059-ba1e-4b23-995c-6cf53f616df5","Type":"ContainerStarted","Data":"f1411433fe3db919dab44913f1175e5f8e716ca735078a3f1eb3633f2e4c5e03"}
Jul 07 01:32:12.753236 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:12.753203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp" event={"ID":"9695e059-ba1e-4b23-995c-6cf53f616df5","Type":"ContainerStarted","Data":"81da478374690be4428491dd7573fb758231b8a017943dba79c48d374c89f1c2"}
Jul 07 01:32:12.753659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:12.753285    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:12.770150 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:12.770098    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp" podStartSLOduration=2.504810896 podStartE2EDuration="4.770079888s" totalImagesPullingTime="2.265268992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:32:12.768618078 +0000 UTC m=+1861.043609401" watchObservedRunningTime="2026-07-07 01:32:12.770079888 +0000 UTC m=+1861.045071210"
Jul 07 01:32:13.152306 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:13.152211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:13.342195 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:13.342157    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:32:13.792110 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:13.792077    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:13.892218 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:13.892187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:14.214062 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:14.213969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:18.763600 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:18.763568    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:19.928957 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:19.928915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:20.029431 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.029401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:20.338838 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.338734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:20.631858 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.631765    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"]
Jul 07 01:32:20.632032 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.631985    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp" podUID="9695e059-ba1e-4b23-995c-6cf53f616df5" containerName="maas-api" containerID="cri-o://81da478374690be4428491dd7573fb758231b8a017943dba79c48d374c89f1c2" gracePeriod=30
Jul 07 01:32:20.662825 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.662794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:20.733793 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.733762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:20.794865 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.794830    2562 generic.go:356] "Generic (PLEG): container finished" podID="9695e059-ba1e-4b23-995c-6cf53f616df5" containerID="81da478374690be4428491dd7573fb758231b8a017943dba79c48d374c89f1c2" exitCode=0
Jul 07 01:32:20.795040 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.794901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp" event={"ID":"9695e059-ba1e-4b23-995c-6cf53f616df5","Type":"ContainerDied","Data":"81da478374690be4428491dd7573fb758231b8a017943dba79c48d374c89f1c2"}
Jul 07 01:32:20.894225 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.894129    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:20.977317 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.977277    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzhcg\" (UniqueName: \"kubernetes.io/projected/9695e059-ba1e-4b23-995c-6cf53f616df5-kube-api-access-vzhcg\") pod \"9695e059-ba1e-4b23-995c-6cf53f616df5\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") "
Jul 07 01:32:20.977317 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.977322    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls\") pod \"9695e059-ba1e-4b23-995c-6cf53f616df5\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") "
Jul 07 01:32:20.977933 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.977364    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695e059-ba1e-4b23-995c-6cf53f616df5-openshift-service-ca\") pod \"9695e059-ba1e-4b23-995c-6cf53f616df5\" (UID: \"9695e059-ba1e-4b23-995c-6cf53f616df5\") "
Jul 07 01:32:20.977933 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.977816    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9695e059-ba1e-4b23-995c-6cf53f616df5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9695e059-ba1e-4b23-995c-6cf53f616df5" (UID: "9695e059-ba1e-4b23-995c-6cf53f616df5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:32:20.979777 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.979751    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9695e059-ba1e-4b23-995c-6cf53f616df5-kube-api-access-vzhcg" (OuterVolumeSpecName: "kube-api-access-vzhcg") pod "9695e059-ba1e-4b23-995c-6cf53f616df5" (UID: "9695e059-ba1e-4b23-995c-6cf53f616df5"). InnerVolumeSpecName "kube-api-access-vzhcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:32:20.979890 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:20.979870    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9695e059-ba1e-4b23-995c-6cf53f616df5" (UID: "9695e059-ba1e-4b23-995c-6cf53f616df5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:32:21.078714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.078636    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzhcg\" (UniqueName: \"kubernetes.io/projected/9695e059-ba1e-4b23-995c-6cf53f616df5-kube-api-access-vzhcg\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:21.078714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.078671    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9695e059-ba1e-4b23-995c-6cf53f616df5-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:21.078714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.078720    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695e059-ba1e-4b23-995c-6cf53f616df5-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:21.800982 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.800884    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"
Jul 07 01:32:21.800982 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.800923    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp" event={"ID":"9695e059-ba1e-4b23-995c-6cf53f616df5","Type":"ContainerDied","Data":"f1411433fe3db919dab44913f1175e5f8e716ca735078a3f1eb3633f2e4c5e03"}
Jul 07 01:32:21.801257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.800994    2562 scope.go:117] "RemoveContainer" containerID="81da478374690be4428491dd7573fb758231b8a017943dba79c48d374c89f1c2"
Jul 07 01:32:21.802631 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.802610    2562 generic.go:356] "Generic (PLEG): container finished" podID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerID="b20069e74826c8f9a55a36cbe83f8ed6f2e8620b30e2b7ba865208120ae1c30c" exitCode=0
Jul 07 01:32:21.802761 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.802648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" event={"ID":"4f39b681-516c-4208-8f9c-f944d2b91ed5","Type":"ContainerDied","Data":"b20069e74826c8f9a55a36cbe83f8ed6f2e8620b30e2b7ba865208120ae1c30c"}
Jul 07 01:32:21.803062 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.803048    2562 scope.go:117] "RemoveContainer" containerID="b20069e74826c8f9a55a36cbe83f8ed6f2e8620b30e2b7ba865208120ae1c30c"
Jul 07 01:32:21.804090 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.804058    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:21.804181 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.804150    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:21.804230 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.804199    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:21.842179 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.842137    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"]
Jul 07 01:32:21.847896 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:21.847863    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-4a5d2769-7bb8c44ff5-mrcxp"]
Jul 07 01:32:22.289397 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:22.289360    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9695e059-ba1e-4b23-995c-6cf53f616df5" path="/var/lib/kubelet/pods/9695e059-ba1e-4b23-995c-6cf53f616df5/volumes"
Jul 07 01:32:22.722820 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:22.722762    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:22.809292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:22.809258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" event={"ID":"4f39b681-516c-4208-8f9c-f944d2b91ed5","Type":"ContainerStarted","Data":"9226c814207a4351fb3f10cd0e72bdee32313353f90376aa9e528c99658f4c08"}
Jul 07 01:32:23.243705 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:23.243647    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:23.244732 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:23.244699    2562 patch_prober.go:28] interesting pod/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 01:32:23.244856 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:23.244782    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 01:32:24.244216 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:24.244174    2562 patch_prober.go:28] interesting pod/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 01:32:24.244737 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:24.244244    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 01:32:24.355151 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:24.355115    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:32:25.244086 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:25.244044    2562 patch_prober.go:28] interesting pod/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 01:32:25.244338 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:25.244113    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 01:32:26.243847 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.243799    2562 patch_prober.go:28] interesting pod/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 01:32:26.244076 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.243856    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 01:32:26.396763 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.396666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:26.456420 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.456376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:26.639907 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.639810    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"]
Jul 07 01:32:26.640360 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.640326    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy" containerID="cri-o://9226c814207a4351fb3f10cd0e72bdee32313353f90376aa9e528c99658f4c08" gracePeriod=30
Jul 07 01:32:26.651377 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.651343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:26.912276 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:26.912179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:27.012429 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:27.012393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:27.115524 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:27.115490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:27.212805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:27.212719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:31.857967 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:31.857936    2562 generic.go:356] "Generic (PLEG): container finished" podID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerID="9226c814207a4351fb3f10cd0e72bdee32313353f90376aa9e528c99658f4c08" exitCode=0
Jul 07 01:32:31.858353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:31.858018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" event={"ID":"4f39b681-516c-4208-8f9c-f944d2b91ed5","Type":"ContainerDied","Data":"9226c814207a4351fb3f10cd0e72bdee32313353f90376aa9e528c99658f4c08"}
Jul 07 01:32:31.858353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:31.858084    2562 scope.go:117] "RemoveContainer" containerID="b20069e74826c8f9a55a36cbe83f8ed6f2e8620b30e2b7ba865208120ae1c30c"
Jul 07 01:32:32.004111 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.004085    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:32.083651 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083554    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-podinfo\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083651 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083613    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39b681-516c-4208-8f9c-f944d2b91ed5-istiod-ca-cert\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083651 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-data\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083672    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-token\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-envoy\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083766    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-certs\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-socket\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083855    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq6c8\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-kube-api-access-lq6c8\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.083946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.083897    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-credential-socket\") pod \"4f39b681-516c-4208-8f9c-f944d2b91ed5\" (UID: \"4f39b681-516c-4208-8f9c-f944d2b91ed5\") "
Jul 07 01:32:32.084266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.084088    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:32:32.084266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.084156    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:32:32.084266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.084103    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f39b681-516c-4208-8f9c-f944d2b91ed5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:32:32.084431 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.084325    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:32:32.084749 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.084720    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:32:32.086016 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.085992    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:32:32.086216 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.086200    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-kube-api-access-lq6c8" (OuterVolumeSpecName: "kube-api-access-lq6c8") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "kube-api-access-lq6c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:32:32.086561 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.086544    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:32:32.086628 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.086554    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f39b681-516c-4208-8f9c-f944d2b91ed5" (UID: "4f39b681-516c-4208-8f9c-f944d2b91ed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:32:32.185224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185187    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185218    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185227    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f39b681-516c-4208-8f9c-f944d2b91ed5-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185235    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185494 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185242    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185494 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185250    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185494 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185257    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185494 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185264    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f39b681-516c-4208-8f9c-f944d2b91ed5-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.185494 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.185272    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6c8\" (UniqueName: \"kubernetes.io/projected/4f39b681-516c-4208-8f9c-f944d2b91ed5-kube-api-access-lq6c8\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:32:32.726573 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.726534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:32.784572 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.784538    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:32.869562 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.869527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:32.871885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.871859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h" event={"ID":"4f39b681-516c-4208-8f9c-f944d2b91ed5","Type":"ContainerDied","Data":"19759a5afc7e35d1b79d807a62a89718c9a0109db4f34e3fbeb318873088c609"}
Jul 07 01:32:32.871885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.871876    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"
Jul 07 01:32:32.872076 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.871941    2562 scope.go:117] "RemoveContainer" containerID="9226c814207a4351fb3f10cd0e72bdee32313353f90376aa9e528c99658f4c08"
Jul 07 01:32:32.889377 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.889349    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"]
Jul 07 01:32:32.892638 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.892608    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4a5d2769-openshift-default-596c6478c9-p6j6h"]
Jul 07 01:32:32.955181 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:32.955146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:33.087487 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:33.087401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:34.289812 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:34.289713    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" path="/var/lib/kubelet/pods/4f39b681-516c-4208-8f9c-f944d2b91ed5/volumes"
Jul 07 01:32:38.626593 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:38.626552    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:38.683381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:38.683344    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:45.322823 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.322790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:45.609176 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.609079    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"]
Jul 07 01:32:45.609888 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.609865    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9695e059-ba1e-4b23-995c-6cf53f616df5" containerName="maas-api"
Jul 07 01:32:45.610030 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.610013    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9695e059-ba1e-4b23-995c-6cf53f616df5" containerName="maas-api"
Jul 07 01:32:45.610115 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.610047    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy"
Jul 07 01:32:45.610115 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.610057    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy"
Jul 07 01:32:45.610226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.610157    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy"
Jul 07 01:32:45.610226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.610168    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy"
Jul 07 01:32:45.610226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.610179    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9695e059-ba1e-4b23-995c-6cf53f616df5" containerName="maas-api"
Jul 07 01:32:45.613257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.613235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.615951 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.615930    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-685d77ff-openshift-default-dockercfg-fk2p9\""
Jul 07 01:32:45.616062 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.615977    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:32:45.625086 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.625059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"]
Jul 07 01:32:45.636082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.636051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:45.707870 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.707830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-socket\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.707870 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.707868    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-data\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.708088 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.707898    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-envoy\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.708088 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.707920    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b023779-e1c3-450d-8984-4f5b716ab896-istio-podinfo\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.708088 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.707949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-istio-token\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.708088 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.707986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b023779-e1c3-450d-8984-4f5b716ab896-istiod-ca-cert\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.708233 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.708096    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-certs\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.708233 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.708132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-credential-socket\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.708233 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.708209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5xx\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-kube-api-access-dh5xx\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.809660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-credential-socket\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.809879 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5xx\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-kube-api-access-dh5xx\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.809879 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-socket\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.809879 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-data\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.809879 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-envoy\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.809879 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b023779-e1c3-450d-8984-4f5b716ab896-istio-podinfo\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.809879 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809878    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-istio-token\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.810194 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b023779-e1c3-450d-8984-4f5b716ab896-istiod-ca-cert\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.810194 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.809951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-certs\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.810194 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.810121    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-credential-socket\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.810482 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.810460    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-data\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.810715 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.810531    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-socket\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.810819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.810563    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-certs\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.811214 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.811191    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b023779-e1c3-450d-8984-4f5b716ab896-istiod-ca-cert\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.812506 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.812475    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-envoy\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.812737 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.812715    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b023779-e1c3-450d-8984-4f5b716ab896-istio-podinfo\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.823513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.823474    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5xx\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-kube-api-access-dh5xx\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.823703 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.823530    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-istio-token\") pod \"e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") " pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:45.925067 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:45.924978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:46.078660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.078633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"]
Jul 07 01:32:46.081866 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:32:46.081823    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b023779_e1c3_450d_8984_4f5b716ab896.slice/crio-fb39f2c97110e410a3a375c9d90432512d506c0de9e26a7e125eef33552d0961 WatchSource:0}: Error finding container fb39f2c97110e410a3a375c9d90432512d506c0de9e26a7e125eef33552d0961: Status 404 returned error can't find the container with id fb39f2c97110e410a3a375c9d90432512d506c0de9e26a7e125eef33552d0961
Jul 07 01:32:46.083913 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.083867    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:46.083975 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.083953    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:46.084010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.083983    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:32:46.087882 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.087857    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:46.939728 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.939690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" event={"ID":"0b023779-e1c3-450d-8984-4f5b716ab896","Type":"ContainerStarted","Data":"db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2"}
Jul 07 01:32:46.940206 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.939747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" event={"ID":"0b023779-e1c3-450d-8984-4f5b716ab896","Type":"ContainerStarted","Data":"fb39f2c97110e410a3a375c9d90432512d506c0de9e26a7e125eef33552d0961"}
Jul 07 01:32:46.960184 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:46.960138    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" podStartSLOduration=1.960121349 podStartE2EDuration="1.960121349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:32:46.958353547 +0000 UTC m=+1895.233344868" watchObservedRunningTime="2026-07-07 01:32:46.960121349 +0000 UTC m=+1895.235112728"
Jul 07 01:32:47.925460 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:47.925413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:47.930610 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:47.930582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:47.945984 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:47.945960    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:47.946871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:47.946850    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:32:48.915710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:48.915650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:49.026825 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.026791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:49.160403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.160368    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"]
Jul 07 01:32:49.160766 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.160752    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy"
Jul 07 01:32:49.160766 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.160767    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f39b681-516c-4208-8f9c-f944d2b91ed5" containerName="istio-proxy"
Jul 07 01:32:49.163916 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.163896    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.166818 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.166745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-685d77ff-serving-cert\""
Jul 07 01:32:49.166952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.166814    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jtxhz\""
Jul 07 01:32:49.172954 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.172925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"]
Jul 07 01:32:49.195844 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.195796    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:49.240203 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.240159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.240415 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.240218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcm4l\" (UniqueName: \"kubernetes.io/projected/ea7aef6e-8340-458e-a527-68e777a98827-kube-api-access-lcm4l\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.240415 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.240302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7aef6e-8340-458e-a527-68e777a98827-openshift-service-ca\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.340647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.340608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.340647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.340654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcm4l\" (UniqueName: \"kubernetes.io/projected/ea7aef6e-8340-458e-a527-68e777a98827-kube-api-access-lcm4l\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.341112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.340755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7aef6e-8340-458e-a527-68e777a98827-openshift-service-ca\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.341112 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:32:49.340783    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-685d77ff-serving-cert: secret "maas-api-e2e-mt-685d77ff-serving-cert" not found
Jul 07 01:32:49.341112 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:32:49.340880    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls podName:ea7aef6e-8340-458e-a527-68e777a98827 nodeName:}" failed. No retries permitted until 2026-07-07 01:32:49.840860491 +0000 UTC m=+1898.115851792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls") pod "maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj" (UID: "ea7aef6e-8340-458e-a527-68e777a98827") : secret "maas-api-e2e-mt-685d77ff-serving-cert" not found
Jul 07 01:32:49.341414 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.341395    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7aef6e-8340-458e-a527-68e777a98827-openshift-service-ca\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.349312 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.349280    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcm4l\" (UniqueName: \"kubernetes.io/projected/ea7aef6e-8340-458e-a527-68e777a98827-kube-api-access-lcm4l\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.432923 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.432823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:49.596157 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.596119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:49.700024 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.699935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:49.843974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.843936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:49.846526 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:49.846498    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls\") pod \"maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") " pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:50.077633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:50.077591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:50.217602 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:50.217571    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"]
Jul 07 01:32:50.220280 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:32:50.220251    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7aef6e_8340_458e_a527_68e777a98827.slice/crio-9b92fbd07e5ec6b93e9c0feed9e28efe121d0ab31828debceec194b987969b88 WatchSource:0}: Error finding container 9b92fbd07e5ec6b93e9c0feed9e28efe121d0ab31828debceec194b987969b88: Status 404 returned error can't find the container with id 9b92fbd07e5ec6b93e9c0feed9e28efe121d0ab31828debceec194b987969b88
Jul 07 01:32:50.960828 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:50.960718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj" event={"ID":"ea7aef6e-8340-458e-a527-68e777a98827","Type":"ContainerStarted","Data":"b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285"}
Jul 07 01:32:50.960828 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:50.960780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj" event={"ID":"ea7aef6e-8340-458e-a527-68e777a98827","Type":"ContainerStarted","Data":"9b92fbd07e5ec6b93e9c0feed9e28efe121d0ab31828debceec194b987969b88"}
Jul 07 01:32:50.960828 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:50.960810    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:32:50.976730 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:50.976653    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj" podStartSLOduration=1.645345509 podStartE2EDuration="1.976633839s" totalImagesPullingTime="331.28833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:32:50.975136273 +0000 UTC m=+1899.250127598" watchObservedRunningTime="2026-07-07 01:32:50.976633839 +0000 UTC m=+1899.251625162"
Jul 07 01:32:53.334738 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:53.334707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:32:53.541469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:53.541429    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:32:56.971746 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:32:56.971720    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:33:02.013860 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:02.013820    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b023779-e1c3-450d-8984-4f5b716ab896" containerID="db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2" exitCode=0
Jul 07 01:33:02.013860 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:02.013866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" event={"ID":"0b023779-e1c3-450d-8984-4f5b716ab896","Type":"ContainerDied","Data":"db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2"}
Jul 07 01:33:02.014340 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:02.014235    2562 scope.go:117] "RemoveContainer" containerID="db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2"
Jul 07 01:33:02.015228 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:02.015200    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:33:02.015289 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:02.015270    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:33:02.015335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:02.015302    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:33:02.946638 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:02.946591    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:33:03.022013 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.021973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" event={"ID":"0b023779-e1c3-450d-8984-4f5b716ab896","Type":"ContainerStarted","Data":"2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2"}
Jul 07 01:33:03.598275 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.598240    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"]
Jul 07 01:33:03.601712 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.601694    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.604358 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.604332    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-685d77ff\"/\"openshift-service-ca.crt\""
Jul 07 01:33:03.604551 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.604361    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-685d77ff\"/\"kube-root-ca.crt\""
Jul 07 01:33:03.604551 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.604361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-685d77ff\"/\"default-dockercfg-b5xbs\""
Jul 07 01:33:03.605400 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.605385    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-685d77ff\"/\"e2e-lifecycle-model-685d77ff-kserve-self-signed-certs\""
Jul 07 01:33:03.610582 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.610560    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"]
Jul 07 01:33:03.631325 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.631278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:03.663186 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.663146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a34339-5930-45ae-8298-ca96dada574f-tls-certs\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.663373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.663198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-model-cache\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.663373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.663260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-dshm\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.663373 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.663289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-tmp-dir\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.663478 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.663385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-home\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.663478 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.663414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7p7\" (UniqueName: \"kubernetes.io/projected/37a34339-5930-45ae-8298-ca96dada574f-kube-api-access-gk7p7\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764255 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a34339-5930-45ae-8298-ca96dada574f-tls-certs\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764255 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-model-cache\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764483 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-dshm\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764483 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-tmp-dir\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764593 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-home\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764593 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764531    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7p7\" (UniqueName: \"kubernetes.io/projected/37a34339-5930-45ae-8298-ca96dada574f-kube-api-access-gk7p7\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764746 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764636    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-model-cache\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.764887 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764856    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-tmp-dir\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.765011 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.764884    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-home\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.766921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.766880    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-dshm\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.767217 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.767200    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a34339-5930-45ae-8298-ca96dada574f-tls-certs\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.772338 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.772318    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7p7\" (UniqueName: \"kubernetes.io/projected/37a34339-5930-45ae-8298-ca96dada574f-kube-api-access-gk7p7\") pod \"e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") " pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.912803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.912678    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:03.925969 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.925873    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:33:03.929299 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.928928    2562 patch_prober.go:28] interesting pod/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 07 01:33:03.929299 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:03.928989    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 07 01:33:04.106493 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:33:04.106460    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37a34339_5930_45ae_8298_ca96dada574f.slice/crio-eb7ef075f476458ae6521baeb0fd03d695cbea301be758b31f9113050895a10e WatchSource:0}: Error finding container eb7ef075f476458ae6521baeb0fd03d695cbea301be758b31f9113050895a10e: Status 404 returned error can't find the container with id eb7ef075f476458ae6521baeb0fd03d695cbea301be758b31f9113050895a10e
Jul 07 01:33:04.107384 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:04.107355    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"]
Jul 07 01:33:04.790831 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:04.790777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:04.891155 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:04.891126    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:04.925971 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:04.925907    2562 patch_prober.go:28] interesting pod/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 07 01:33:04.926250 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:04.926013    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 07 01:33:05.060744 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:05.060613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z" event={"ID":"37a34339-5930-45ae-8298-ca96dada574f","Type":"ContainerStarted","Data":"88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6"}
Jul 07 01:33:05.060744 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:05.060702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z" event={"ID":"37a34339-5930-45ae-8298-ca96dada574f","Type":"ContainerStarted","Data":"eb7ef075f476458ae6521baeb0fd03d695cbea301be758b31f9113050895a10e"}
Jul 07 01:33:05.076262 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:05.076177    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z" podStartSLOduration=2.076162133 podStartE2EDuration="2.076162133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:33:05.07532929 +0000 UTC m=+1913.350320612" watchObservedRunningTime="2026-07-07 01:33:05.076162133 +0000 UTC m=+1913.351153455"
Jul 07 01:33:05.926180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:05.925703    2562 patch_prober.go:28] interesting pod/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 07 01:33:05.926180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:05.925770    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 07 01:33:06.544225 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:06.544179    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:33:06.925987 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:06.925880    2562 patch_prober.go:28] interesting pod/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 07 01:33:06.926290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:06.926255    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 07 01:33:07.929589 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:07.929560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:33:07.930091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:07.929845    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:33:07.930722 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:07.930705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:33:11.005970 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:11.005929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:12.381299 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:12.381263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:13.912887 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:13.912851    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:13.913303 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:13.912904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:13.925413 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:13.925379    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:14.114766 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:14.114738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:18.146660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:18.146616    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:33:20.526594 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:20.526561    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:20.804548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:20.804443    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"]
Jul 07 01:33:20.804769 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:20.804649    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj" podUID="ea7aef6e-8340-458e-a527-68e777a98827" containerName="maas-api" containerID="cri-o://b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285" gracePeriod=30
Jul 07 01:33:20.834843 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:20.834812    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:21.055718 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.055635    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:33:21.123561 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.123513    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7aef6e-8340-458e-a527-68e777a98827-openshift-service-ca\") pod \"ea7aef6e-8340-458e-a527-68e777a98827\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") "
Jul 07 01:33:21.123800 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.123627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcm4l\" (UniqueName: \"kubernetes.io/projected/ea7aef6e-8340-458e-a527-68e777a98827-kube-api-access-lcm4l\") pod \"ea7aef6e-8340-458e-a527-68e777a98827\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") "
Jul 07 01:33:21.123800 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.123735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls\") pod \"ea7aef6e-8340-458e-a527-68e777a98827\" (UID: \"ea7aef6e-8340-458e-a527-68e777a98827\") "
Jul 07 01:33:21.124031 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.124005    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7aef6e-8340-458e-a527-68e777a98827-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea7aef6e-8340-458e-a527-68e777a98827" (UID: "ea7aef6e-8340-458e-a527-68e777a98827"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:33:21.126108 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.126075    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea7aef6e-8340-458e-a527-68e777a98827" (UID: "ea7aef6e-8340-458e-a527-68e777a98827"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:33:21.126215 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.126158    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7aef6e-8340-458e-a527-68e777a98827-kube-api-access-lcm4l" (OuterVolumeSpecName: "kube-api-access-lcm4l") pod "ea7aef6e-8340-458e-a527-68e777a98827" (UID: "ea7aef6e-8340-458e-a527-68e777a98827"). InnerVolumeSpecName "kube-api-access-lcm4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:33:21.136581 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.136547    2562 generic.go:356] "Generic (PLEG): container finished" podID="ea7aef6e-8340-458e-a527-68e777a98827" containerID="b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285" exitCode=0
Jul 07 01:33:21.136760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.136647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj" event={"ID":"ea7aef6e-8340-458e-a527-68e777a98827","Type":"ContainerDied","Data":"b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285"}
Jul 07 01:33:21.136760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.136662    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"
Jul 07 01:33:21.136760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.136735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj" event={"ID":"ea7aef6e-8340-458e-a527-68e777a98827","Type":"ContainerDied","Data":"9b92fbd07e5ec6b93e9c0feed9e28efe121d0ab31828debceec194b987969b88"}
Jul 07 01:33:21.136760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.136757    2562 scope.go:117] "RemoveContainer" containerID="b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285"
Jul 07 01:33:21.146045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.146020    2562 scope.go:117] "RemoveContainer" containerID="b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285"
Jul 07 01:33:21.146361 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:33:21.146339    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285\": container with ID starting with b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285 not found: ID does not exist" containerID="b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285"
Jul 07 01:33:21.146421 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.146371    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285"} err="failed to get container status \"b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285\": rpc error: code = NotFound desc = could not find container \"b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285\": container with ID starting with b5a641bad0b7776df1e7abbb3d2a2f4be152651524b72fff16ac163c422d0285 not found: ID does not exist"
Jul 07 01:33:21.164860 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.164829    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"]
Jul 07 01:33:21.170452 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.170420    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-685d77ff-cc46f6976-vsqlj"]
Jul 07 01:33:21.224779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.224742    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcm4l\" (UniqueName: \"kubernetes.io/projected/ea7aef6e-8340-458e-a527-68e777a98827-kube-api-access-lcm4l\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:21.224779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.224771    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7aef6e-8340-458e-a527-68e777a98827-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:21.224779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:21.224783    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7aef6e-8340-458e-a527-68e777a98827-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:22.289462 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:22.289428    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7aef6e-8340-458e-a527-68e777a98827" path="/var/lib/kubelet/pods/ea7aef6e-8340-458e-a527-68e777a98827/volumes"
Jul 07 01:33:26.576626 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:26.576589    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:29.433400 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:29.433361    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:33:31.569751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:31.569715    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"]
Jul 07 01:33:31.609363 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:31.609322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:33:46.591207 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.591165    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z" podUID="37a34339-5930-45ae-8298-ca96dada574f" containerName="main" containerID="cri-o://88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6" gracePeriod=45
Jul 07 01:33:46.854257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.854185    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:46.944430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-dshm\") pod \"37a34339-5930-45ae-8298-ca96dada574f\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") "
Jul 07 01:33:46.944430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944436    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a34339-5930-45ae-8298-ca96dada574f-tls-certs\") pod \"37a34339-5930-45ae-8298-ca96dada574f\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") "
Jul 07 01:33:46.944709 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944474    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-tmp-dir\") pod \"37a34339-5930-45ae-8298-ca96dada574f\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") "
Jul 07 01:33:46.944709 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-home\") pod \"37a34339-5930-45ae-8298-ca96dada574f\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") "
Jul 07 01:33:46.944709 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk7p7\" (UniqueName: \"kubernetes.io/projected/37a34339-5930-45ae-8298-ca96dada574f-kube-api-access-gk7p7\") pod \"37a34339-5930-45ae-8298-ca96dada574f\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") "
Jul 07 01:33:46.944709 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944568    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-model-cache\") pod \"37a34339-5930-45ae-8298-ca96dada574f\" (UID: \"37a34339-5930-45ae-8298-ca96dada574f\") "
Jul 07 01:33:46.944929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944775    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-home" (OuterVolumeSpecName: "home") pod "37a34339-5930-45ae-8298-ca96dada574f" (UID: "37a34339-5930-45ae-8298-ca96dada574f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:33:46.944929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944793    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37a34339-5930-45ae-8298-ca96dada574f" (UID: "37a34339-5930-45ae-8298-ca96dada574f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:33:46.945018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944939    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-model-cache" (OuterVolumeSpecName: "model-cache") pod "37a34339-5930-45ae-8298-ca96dada574f" (UID: "37a34339-5930-45ae-8298-ca96dada574f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:33:46.945018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944965    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:46.945018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.944983    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:46.946979 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.946955    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-dshm" (OuterVolumeSpecName: "dshm") pod "37a34339-5930-45ae-8298-ca96dada574f" (UID: "37a34339-5930-45ae-8298-ca96dada574f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:33:46.947072 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.946977    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a34339-5930-45ae-8298-ca96dada574f-kube-api-access-gk7p7" (OuterVolumeSpecName: "kube-api-access-gk7p7") pod "37a34339-5930-45ae-8298-ca96dada574f" (UID: "37a34339-5930-45ae-8298-ca96dada574f"). InnerVolumeSpecName "kube-api-access-gk7p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:33:46.947072 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:46.947047    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a34339-5930-45ae-8298-ca96dada574f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37a34339-5930-45ae-8298-ca96dada574f" (UID: "37a34339-5930-45ae-8298-ca96dada574f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:33:47.045613 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.045569    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk7p7\" (UniqueName: \"kubernetes.io/projected/37a34339-5930-45ae-8298-ca96dada574f-kube-api-access-gk7p7\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:47.045613 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.045605    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:47.045613 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.045614    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a34339-5930-45ae-8298-ca96dada574f-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:47.045613 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.045624    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a34339-5930-45ae-8298-ca96dada574f-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:33:47.260741 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.260664    2562 generic.go:356] "Generic (PLEG): container finished" podID="37a34339-5930-45ae-8298-ca96dada574f" containerID="88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6" exitCode=0
Jul 07 01:33:47.260741 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.260736    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"
Jul 07 01:33:47.260997 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.260758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z" event={"ID":"37a34339-5930-45ae-8298-ca96dada574f","Type":"ContainerDied","Data":"88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6"}
Jul 07 01:33:47.260997 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.260801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z" event={"ID":"37a34339-5930-45ae-8298-ca96dada574f","Type":"ContainerDied","Data":"eb7ef075f476458ae6521baeb0fd03d695cbea301be758b31f9113050895a10e"}
Jul 07 01:33:47.260997 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.260828    2562 scope.go:117] "RemoveContainer" containerID="88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6"
Jul 07 01:33:47.270468 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.270447    2562 scope.go:117] "RemoveContainer" containerID="88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6"
Jul 07 01:33:47.270790 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:33:47.270770    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6\": container with ID starting with 88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6 not found: ID does not exist" containerID="88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6"
Jul 07 01:33:47.270871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.270802    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6"} err="failed to get container status \"88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6\": rpc error: code = NotFound desc = could not find container \"88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6\": container with ID starting with 88c05909d914276dbbac340f3df5e06664c5332650c601e3a80cbf8b53a999c6 not found: ID does not exist"
Jul 07 01:33:47.281105 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.281074    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"]
Jul 07 01:33:47.284819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:47.284792    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-685d77ff/e2e-lifecycle-model-685d77ff-kserve-5df9d4844c-78s5z"]
Jul 07 01:33:48.289863 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:33:48.289816    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37a34339-5930-45ae-8298-ca96dada574f" path="/var/lib/kubelet/pods/37a34339-5930-45ae-8298-ca96dada574f/volumes"
Jul 07 01:34:07.770894 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:07.770804    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:08.597701 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:08.597654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:10.140267 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:10.140230    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:10.242693 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:10.242643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:11.142434 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:11.142373    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:34:14.024001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:14.023969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:19.797936 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:19.797899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:19.856635 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:19.856584    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:20.051518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:20.051431    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"]
Jul 07 01:34:20.051756 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:20.051728    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy" containerID="cri-o://2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2" gracePeriod=30
Jul 07 01:34:20.071185 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:20.071147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:20.302762 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:20.302664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:21.342908 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:21.342869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:21.443032 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:21.442998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:21.543298 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:21.543260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:22.292579 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:22.292549    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:22.401957 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:22.401928    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:22.932660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:22.932618    2562 patch_prober.go:28] interesting pod/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 01:34:22.932843 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:22.932698    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 01:34:23.752768 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:23.752729    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:34:23.790603 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:23.790568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:25.317027 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.316952    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:34:25.443873 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.443841    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b023779-e1c3-450d-8984-4f5b716ab896" containerID="2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2" exitCode=0
Jul 07 01:34:25.444077 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.443913    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"
Jul 07 01:34:25.444077 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.443916    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" event={"ID":"0b023779-e1c3-450d-8984-4f5b716ab896","Type":"ContainerDied","Data":"2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2"}
Jul 07 01:34:25.444077 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.443978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs" event={"ID":"0b023779-e1c3-450d-8984-4f5b716ab896","Type":"ContainerDied","Data":"fb39f2c97110e410a3a375c9d90432512d506c0de9e26a7e125eef33552d0961"}
Jul 07 01:34:25.444077 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.444010    2562 scope.go:117] "RemoveContainer" containerID="2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2"
Jul 07 01:34:25.452995 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.452976    2562 scope.go:117] "RemoveContainer" containerID="db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2"
Jul 07 01:34:25.461485 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.461464    2562 scope.go:117] "RemoveContainer" containerID="2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2"
Jul 07 01:34:25.461840 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:34:25.461811    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2\": container with ID starting with 2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2 not found: ID does not exist" containerID="2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2"
Jul 07 01:34:25.461945 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.461846    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2"} err="failed to get container status \"2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2\": rpc error: code = NotFound desc = could not find container \"2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2\": container with ID starting with 2da4b29bda3e18df1d851d290d6888f07edf3e227c2e70f3bc0de217895d7fb2 not found: ID does not exist"
Jul 07 01:34:25.461945 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.461869    2562 scope.go:117] "RemoveContainer" containerID="db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2"
Jul 07 01:34:25.462237 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:34:25.462214    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2\": container with ID starting with db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2 not found: ID does not exist" containerID="db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2"
Jul 07 01:34:25.462334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.462243    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2"} err="failed to get container status \"db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2\": rpc error: code = NotFound desc = could not find container \"db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2\": container with ID starting with db1d5311c7351c98b94a93a717154ac48d7b6b6ecee43e4b147a522582f83ba2 not found: ID does not exist"
Jul 07 01:34:25.463450 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463428    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh5xx\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-kube-api-access-dh5xx\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463555 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463470    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-envoy\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463555 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463491    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-certs\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463555 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463544    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-data\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463749 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463560    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-credential-socket\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463749 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463601    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-socket\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463749 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463650    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b023779-e1c3-450d-8984-4f5b716ab896-istiod-ca-cert\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463749 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463712    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-istio-token\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463750    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b023779-e1c3-450d-8984-4f5b716ab896-istio-podinfo\") pod \"0b023779-e1c3-450d-8984-4f5b716ab896\" (UID: \"0b023779-e1c3-450d-8984-4f5b716ab896\") "
Jul 07 01:34:25.463952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463872    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:34:25.463952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.463889    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:34:25.464099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.464079    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.464160 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.464099    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.464217 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.464192    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b023779-e1c3-450d-8984-4f5b716ab896-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:34:25.464458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.464413    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:34:25.464758 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.464586    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:34:25.466033 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.466007    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b023779-e1c3-450d-8984-4f5b716ab896-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:34:25.466247 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.466225    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-kube-api-access-dh5xx" (OuterVolumeSpecName: "kube-api-access-dh5xx") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "kube-api-access-dh5xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:34:25.466369 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.466334    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:34:25.466439 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.466397    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b023779-e1c3-450d-8984-4f5b716ab896" (UID: "0b023779-e1c3-450d-8984-4f5b716ab896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:34:25.565134 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.565094    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.565134 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.565129    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b023779-e1c3-450d-8984-4f5b716ab896-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.565134 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.565138    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh5xx\" (UniqueName: \"kubernetes.io/projected/0b023779-e1c3-450d-8984-4f5b716ab896-kube-api-access-dh5xx\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.565392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.565149    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.565392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.565158    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.565392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.565165    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b023779-e1c3-450d-8984-4f5b716ab896-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.565392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.565173    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b023779-e1c3-450d-8984-4f5b716ab896-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:34:25.769217 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.769186    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"]
Jul 07 01:34:25.772310 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:25.772285    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-685d77ff-openshift-default-75944cccff-kqfjs"]
Jul 07 01:34:26.290097 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:26.290066    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" path="/var/lib/kubelet/pods/0b023779-e1c3-450d-8984-4f5b716ab896/volumes"
Jul 07 01:34:29.359574 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:29.359535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:29.418735 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:29.418675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:30.031445 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:30.031412    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:35.832940 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:35.832908    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:35.888550 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:35.888501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:35.974558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:35.974515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:36.070957 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:36.070921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:37.342419 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:37.342376    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:34:46.593088 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:46.593054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:46.905591 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:46.905494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:34:50.418677 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:34:50.418614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:07.845814 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:07.845765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:11.285606 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:11.285575    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:11.333059 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:11.333017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:13.553446 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:13.553407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:13.653803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:13.653765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:13.888205 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:13.888114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:14.156641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.156555    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"]
Jul 07 01:35:14.156969 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.156949    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a34339-5930-45ae-8298-ca96dada574f" containerName="main"
Jul 07 01:35:14.156969 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.156967    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a34339-5930-45ae-8298-ca96dada574f" containerName="main"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.156981    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.156987    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.157002    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7aef6e-8340-458e-a527-68e777a98827" containerName="maas-api"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.157010    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7aef6e-8340-458e-a527-68e777a98827" containerName="maas-api"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.157026    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.157031    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.157095    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.157105    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a34339-5930-45ae-8298-ca96dada574f" containerName="main"
Jul 07 01:35:14.157169 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.157117    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7aef6e-8340-458e-a527-68e777a98827" containerName="maas-api"
Jul 07 01:35:14.161503 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.161482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.163992 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.163966    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:35:14.164102 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.164034    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-86cc19-openshift-default-dockercfg-dtdft\""
Jul 07 01:35:14.172577 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.172548    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"]
Jul 07 01:35:14.183983 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.183948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:14.263631 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.263595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:14.294271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-podinfo\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-data\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-socket\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-certs\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istiod-ca-cert\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-envoy\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-token\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-credential-socket\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.294677 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.294564    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96cjf\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-kube-api-access-96cjf\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.395646 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.395602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96cjf\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-kube-api-access-96cjf\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.395839 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.395740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-podinfo\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.395839 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.395774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-data\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.395839 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.395817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-socket\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.395870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-certs\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.395902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istiod-ca-cert\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-envoy\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-token\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-credential-socket\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396302    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-data\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396390 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396327    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-socket\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396628 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396380    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-certs\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396759    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-credential-socket\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.396853 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.396823    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istiod-ca-cert\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.399093 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.399071    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-envoy\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.399198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.399112    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-podinfo\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.404774 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.404749    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-token\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.405076 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.405052    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96cjf\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-kube-api-access-96cjf\") pod \"e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") " pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.472973 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.472930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:14.620136 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.620107    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"]
Jul 07 01:35:14.622731 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:35:14.622702    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d88ac5_a6c5_4edd_8095_3d2e17f93507.slice/crio-1ad21fe45107691d77f600d7990cb16a383ea50f845310d20e1d68a4b820c173 WatchSource:0}: Error finding container 1ad21fe45107691d77f600d7990cb16a383ea50f845310d20e1d68a4b820c173: Status 404 returned error can't find the container with id 1ad21fe45107691d77f600d7990cb16a383ea50f845310d20e1d68a4b820c173
Jul 07 01:35:14.625150 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.625113    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:35:14.625285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.625184    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:35:14.625285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.625219    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:35:14.673952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:14.673920    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" event={"ID":"04d88ac5-a6c5-4edd-8095-3d2e17f93507","Type":"ContainerStarted","Data":"1ad21fe45107691d77f600d7990cb16a383ea50f845310d20e1d68a4b820c173"}
Jul 07 01:35:15.680470 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:15.680426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" event={"ID":"04d88ac5-a6c5-4edd-8095-3d2e17f93507","Type":"ContainerStarted","Data":"cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc"}
Jul 07 01:35:15.700057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:15.699985    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podStartSLOduration=1.6999666420000001 podStartE2EDuration="1.699966642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:35:15.698934766 +0000 UTC m=+2043.973926092" watchObservedRunningTime="2026-07-07 01:35:15.699966642 +0000 UTC m=+2043.974957963"
Jul 07 01:35:16.473833 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:16.473804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:16.478861 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:16.478824    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:16.684900 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:16.684860    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:16.685763 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:16.685744    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:17.564382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.564343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:17.663844 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.663804    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"]
Jul 07 01:35:17.664207 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.664195    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b023779-e1c3-450d-8984-4f5b716ab896" containerName="istio-proxy"
Jul 07 01:35:17.667185 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.667166    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.669505 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.669468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:17.669762 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.669740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-86cc19-serving-cert\""
Jul 07 01:35:17.669837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.669763    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jtxhz\""
Jul 07 01:35:17.678399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.678367    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"]
Jul 07 01:35:17.726940 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.726907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1eb203-3029-4e64-92b9-808b32c49ac3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.727454 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.726969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6rp\" (UniqueName: \"kubernetes.io/projected/bc1eb203-3029-4e64-92b9-808b32c49ac3-kube-api-access-mn6rp\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.727454 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.727063    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.752986 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.752936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:17.828017 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.827916    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6rp\" (UniqueName: \"kubernetes.io/projected/bc1eb203-3029-4e64-92b9-808b32c49ac3-kube-api-access-mn6rp\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.828200 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.828076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.828266 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:35:17.828192    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-86cc19-serving-cert: secret "maas-api-e2e-inf-a-86cc19-serving-cert" not found
Jul 07 01:35:17.828266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.828230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1eb203-3029-4e64-92b9-808b32c49ac3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.828375 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:35:17.828275    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls podName:bc1eb203-3029-4e64-92b9-808b32c49ac3 nodeName:}" failed. No retries permitted until 2026-07-07 01:35:18.328250456 +0000 UTC m=+2046.603241758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls") pod "maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv" (UID: "bc1eb203-3029-4e64-92b9-808b32c49ac3") : secret "maas-api-e2e-inf-a-86cc19-serving-cert" not found
Jul 07 01:35:17.828923 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.828899    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1eb203-3029-4e64-92b9-808b32c49ac3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:17.839392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:17.839356    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6rp\" (UniqueName: \"kubernetes.io/projected/bc1eb203-3029-4e64-92b9-808b32c49ac3-kube-api-access-mn6rp\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:18.133439 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.133345    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:18.328263 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.328225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:18.332445 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.332415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:18.335453 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.335417    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") " pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:18.425033 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.424943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:18.579008 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.578976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:18.724675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.724645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"]
Jul 07 01:35:18.727565 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:35:18.727537    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1eb203_3029_4e64_92b9_808b32c49ac3.slice/crio-92bad507665e34817cb333cfce69e75c926f6453c501884ddb454acf41ef4e95 WatchSource:0}: Error finding container 92bad507665e34817cb333cfce69e75c926f6453c501884ddb454acf41ef4e95: Status 404 returned error can't find the container with id 92bad507665e34817cb333cfce69e75c926f6453c501884ddb454acf41ef4e95
Jul 07 01:35:18.729403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:18.729380    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 01:35:19.699568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:19.699532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv" event={"ID":"bc1eb203-3029-4e64-92b9-808b32c49ac3","Type":"ContainerStarted","Data":"2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a"}
Jul 07 01:35:19.699800 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:19.699589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv" event={"ID":"bc1eb203-3029-4e64-92b9-808b32c49ac3","Type":"ContainerStarted","Data":"92bad507665e34817cb333cfce69e75c926f6453c501884ddb454acf41ef4e95"}
Jul 07 01:35:19.699800 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:19.699661    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:19.716021 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:19.715961    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv" podStartSLOduration=2.275164232 podStartE2EDuration="2.715941492s" totalImagesPullingTime="440.77726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:35:19.715017507 +0000 UTC m=+2047.990008830" watchObservedRunningTime="2026-07-07 01:35:19.715941492 +0000 UTC m=+2047.990932791"
Jul 07 01:35:22.056895 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:22.056851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:22.398700 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:22.398567    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:22.696988 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:22.696883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:23.075442 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:23.075350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:24.439850 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:24.439810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:35:25.585518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.585480    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:25.648710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.648110    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"]
Jul 07 01:35:25.652701 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.652626    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.655540 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.655509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-cf6840-serving-cert\""
Jul 07 01:35:25.663211 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.663180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"]
Jul 07 01:35:25.673147 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.673097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:25.709847 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.709820    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:35:25.748307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.748252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:25.799015 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.798957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66149d33-4040-4980-91c8-39d25c0d9e46-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.799241 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.799127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh5nw\" (UniqueName: \"kubernetes.io/projected/66149d33-4040-4980-91c8-39d25c0d9e46-kube-api-access-nh5nw\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.799338 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.799321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.900826 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.900719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66149d33-4040-4980-91c8-39d25c0d9e46-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.900826 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.900807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh5nw\" (UniqueName: \"kubernetes.io/projected/66149d33-4040-4980-91c8-39d25c0d9e46-kube-api-access-nh5nw\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.901074 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.900873    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.901074 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:35:25.900961    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-cf6840-serving-cert: secret "maas-api-e2e-inf-b-cf6840-serving-cert" not found
Jul 07 01:35:25.901074 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:35:25.901015    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls podName:66149d33-4040-4980-91c8-39d25c0d9e46 nodeName:}" failed. No retries permitted until 2026-07-07 01:35:26.400999489 +0000 UTC m=+2054.675990789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls") pod "maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp" (UID: "66149d33-4040-4980-91c8-39d25c0d9e46") : secret "maas-api-e2e-inf-b-cf6840-serving-cert" not found
Jul 07 01:35:25.901495 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.901466    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66149d33-4040-4980-91c8-39d25c0d9e46-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:25.909695 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:25.909658    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh5nw\" (UniqueName: \"kubernetes.io/projected/66149d33-4040-4980-91c8-39d25c0d9e46-kube-api-access-nh5nw\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:26.126880 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:26.126830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:26.225675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:26.225640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:26.406140 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:26.406104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:26.408918 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:26.408881    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") " pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:26.568794 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:26.568667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:26.759806 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:26.759778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"]
Jul 07 01:35:26.762278 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:35:26.762246    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66149d33_4040_4980_91c8_39d25c0d9e46.slice/crio-7b6b8bf89df32cbbe2d12adcb3dce3a38098610e9239e2d9eb4fdd6095033e8b WatchSource:0}: Error finding container 7b6b8bf89df32cbbe2d12adcb3dce3a38098610e9239e2d9eb4fdd6095033e8b: Status 404 returned error can't find the container with id 7b6b8bf89df32cbbe2d12adcb3dce3a38098610e9239e2d9eb4fdd6095033e8b
Jul 07 01:35:27.758403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:27.758364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp" event={"ID":"66149d33-4040-4980-91c8-39d25c0d9e46","Type":"ContainerStarted","Data":"b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470"}
Jul 07 01:35:27.758591 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:27.758464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp" event={"ID":"66149d33-4040-4980-91c8-39d25c0d9e46","Type":"ContainerStarted","Data":"7b6b8bf89df32cbbe2d12adcb3dce3a38098610e9239e2d9eb4fdd6095033e8b"}
Jul 07 01:35:27.758591 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:27.758507    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:27.775209 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:27.775157    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp" podStartSLOduration=2.426102437 podStartE2EDuration="2.775139717s" totalImagesPullingTime="349.03728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:35:27.773114655 +0000 UTC m=+2056.048105977" watchObservedRunningTime="2026-07-07 01:35:27.775139717 +0000 UTC m=+2056.050131038"
Jul 07 01:35:29.976327 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:29.976296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:32.400464 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.400008    2562 patch_prober.go:28] interesting pod/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:59240->10.133.0.46:15021: read: connection reset by peer" start-of-body=
Jul 07 01:35:32.400464 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.400066    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:59240->10.133.0.46:15021: read: connection reset by peer"
Jul 07 01:35:32.784341 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.784302    2562 generic.go:356] "Generic (PLEG): container finished" podID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerID="cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc" exitCode=0
Jul 07 01:35:32.784552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.784360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" event={"ID":"04d88ac5-a6c5-4edd-8095-3d2e17f93507","Type":"ContainerDied","Data":"cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc"}
Jul 07 01:35:32.784839 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.784820    2562 scope.go:117] "RemoveContainer" containerID="cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc"
Jul 07 01:35:32.785924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.785894    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:35:32.785993 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.785970    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:35:32.786030 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:32.786006    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:35:33.768672 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:33.768589    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:35:33.794395 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:33.794359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" event={"ID":"04d88ac5-a6c5-4edd-8095-3d2e17f93507","Type":"ContainerStarted","Data":"9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990"}
Jul 07 01:35:34.473601 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:34.473562    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:34.473852 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:34.473830    2562 patch_prober.go:28] interesting pod/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 07 01:35:34.473904 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:34.473879    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 07 01:35:35.042283 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:35.042238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:35:35.474057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:35.474017    2562 patch_prober.go:28] interesting pod/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 07 01:35:35.474259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:35.474085    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 07 01:35:36.395430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.395397    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"]
Jul 07 01:35:36.399640 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.399620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.402286 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.402260    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86cc19\"/\"kube-root-ca.crt\""
Jul 07 01:35:36.402446 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.402420    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-86cc19\"/\"openshift-service-ca.crt\""
Jul 07 01:35:36.403556 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.403540    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86cc19\"/\"default-dockercfg-zrrsw\""
Jul 07 01:35:36.403619 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.403579    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-86cc19\"/\"test-model-86cc19-kserve-self-signed-certs\""
Jul 07 01:35:36.408842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.408817    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"]
Jul 07 01:35:36.419659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.419626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:36.474324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.474288    2562 patch_prober.go:28] interesting pod/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 07 01:35:36.474529 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.474347    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 07 01:35:36.526426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.526386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tmp-dir\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.526742 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.526718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-dshm\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.526891 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.526875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-model-cache\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.527043 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.527026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-home\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.527183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.527169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tls-certs\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.527342 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.527327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhcd\" (UniqueName: \"kubernetes.io/projected/6bcabd39-11d8-4890-a211-1632c8ca5d1a-kube-api-access-vdhcd\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.628195 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.628125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-home\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.628430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.628235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tls-certs\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.628430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.628314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhcd\" (UniqueName: \"kubernetes.io/projected/6bcabd39-11d8-4890-a211-1632c8ca5d1a-kube-api-access-vdhcd\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.628430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.628346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tmp-dir\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.628430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.628378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-dshm\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.628430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.628406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-model-cache\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.628929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.628902    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-model-cache\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.629335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.629314    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-home\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.630819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.630799    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tmp-dir\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.632992 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.632971    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tls-certs\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.634601 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.634577    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-dshm\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.639901 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.639873    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhcd\" (UniqueName: \"kubernetes.io/projected/6bcabd39-11d8-4890-a211-1632c8ca5d1a-kube-api-access-vdhcd\") pod \"test-model-86cc19-kserve-7477486bf7-x7nq8\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") " pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.710793 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.710674    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:36.896429 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:36.896401    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"]
Jul 07 01:35:36.898383 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:35:36.898253    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bcabd39_11d8_4890_a211_1632c8ca5d1a.slice/crio-d89326b096e374c9462f0360f1d7ca9e072ce686442f023581996a4dec1a102b WatchSource:0}: Error finding container d89326b096e374c9462f0360f1d7ca9e072ce686442f023581996a4dec1a102b: Status 404 returned error can't find the container with id d89326b096e374c9462f0360f1d7ca9e072ce686442f023581996a4dec1a102b
Jul 07 01:35:37.473381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:37.473345    2562 patch_prober.go:28] interesting pod/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 07 01:35:37.473801 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:37.473402    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 07 01:35:37.815633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:37.815530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8" event={"ID":"6bcabd39-11d8-4890-a211-1632c8ca5d1a","Type":"ContainerStarted","Data":"6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336"}
Jul 07 01:35:37.815633 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:37.815585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8" event={"ID":"6bcabd39-11d8-4890-a211-1632c8ca5d1a","Type":"ContainerStarted","Data":"d89326b096e374c9462f0360f1d7ca9e072ce686442f023581996a4dec1a102b"}
Jul 07 01:35:37.832405 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:37.832353    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8" podStartSLOduration=1.832335424 podStartE2EDuration="1.832335424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:35:37.832168367 +0000 UTC m=+2066.107159690" watchObservedRunningTime="2026-07-07 01:35:37.832335424 +0000 UTC m=+2066.107326747"
Jul 07 01:35:38.477706 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:38.477659    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:38.478100 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:38.477971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:38.479056 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:38.479034    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:35:39.826974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:39.826913    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:39.928087 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:39.928001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:44.242519 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:44.242481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:35:46.710846 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:46.710808    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:46.711360 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:46.710863    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:46.725562 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:46.725520    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:46.834378 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:46.834340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:46.871889 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:46.871851    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:35:46.892973 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:46.892934    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:57.169978 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.169932    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"]
Jul 07 01:35:57.173867 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.173843    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.177775 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.177745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf6840\"/\"test-model-cf6840-kserve-self-signed-certs\""
Jul 07 01:35:57.177944 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.177745    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf6840\"/\"openshift-service-ca.crt\""
Jul 07 01:35:57.177944 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.177745    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf6840\"/\"kube-root-ca.crt\""
Jul 07 01:35:57.177944 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.177756    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf6840\"/\"default-dockercfg-4zs4x\""
Jul 07 01:35:57.183386 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.183357    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"]
Jul 07 01:35:57.195746 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.195702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:35:57.220941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.220908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c823efb-f178-4e7f-902f-f29511affa0f-tls-certs\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.221131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.220952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlq2\" (UniqueName: \"kubernetes.io/projected/2c823efb-f178-4e7f-902f-f29511affa0f-kube-api-access-htlq2\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.221131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.221002    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-tmp-dir\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.221131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.221020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-dshm\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.221131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.221047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-model-cache\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.221131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.221091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-home\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.321849 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.321804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-model-cache\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.321849 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.321852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-home\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.322107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.321909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c823efb-f178-4e7f-902f-f29511affa0f-tls-certs\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.322107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.322033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlq2\" (UniqueName: \"kubernetes.io/projected/2c823efb-f178-4e7f-902f-f29511affa0f-kube-api-access-htlq2\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.322107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.322076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-tmp-dir\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.322280 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.322106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-dshm\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.322280 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.322259    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-model-cache\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.322425 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.322405    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-tmp-dir\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.322500 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.322424    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-home\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.324403 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.324385    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-dshm\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.324674 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.324658    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c823efb-f178-4e7f-902f-f29511affa0f-tls-certs\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.330430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.330403    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlq2\" (UniqueName: \"kubernetes.io/projected/2c823efb-f178-4e7f-902f-f29511affa0f-kube-api-access-htlq2\") pod \"test-model-cf6840-kserve-f45b9fb8c-96mbg\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") " pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.485180 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.485136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:35:57.636559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.636522    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"]
Jul 07 01:35:57.640321 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:35:57.640273    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c823efb_f178_4e7f_902f_f29511affa0f.slice/crio-49c0713c99f26a2e77e6fea0b534ac215dfc5532a02843578677efa633567b3d WatchSource:0}: Error finding container 49c0713c99f26a2e77e6fea0b534ac215dfc5532a02843578677efa633567b3d: Status 404 returned error can't find the container with id 49c0713c99f26a2e77e6fea0b534ac215dfc5532a02843578677efa633567b3d
Jul 07 01:35:57.913150 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.913054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg" event={"ID":"2c823efb-f178-4e7f-902f-f29511affa0f","Type":"ContainerStarted","Data":"25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937"}
Jul 07 01:35:57.913150 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.913097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg" event={"ID":"2c823efb-f178-4e7f-902f-f29511affa0f","Type":"ContainerStarted","Data":"49c0713c99f26a2e77e6fea0b534ac215dfc5532a02843578677efa633567b3d"}
Jul 07 01:35:57.929802 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:35:57.929736    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg" podStartSLOduration=0.929714793 podStartE2EDuration="929.714793ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:35:57.928524449 +0000 UTC m=+2086.203515770" watchObservedRunningTime="2026-07-07 01:35:57.929714793 +0000 UTC m=+2086.204706106"
Jul 07 01:36:00.941509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:00.941464    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:36:07.485893 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:07.485846    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:36:07.485893 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:07.485899    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:36:07.498726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:07.498667    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:36:07.973895 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:07.973859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:36:12.318704 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:12.318655    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:36:12.320979 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:12.320959    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:36:13.341143 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:13.341106    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:36:23.284623 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:23.284590    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"]
Jul 07 01:36:23.306031 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:23.305969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:36:24.162022 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.161986    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"]
Jul 07 01:36:24.179206 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.179165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:36:24.326702 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.326631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:36:24.736638 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.736596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:36:24.833044 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.832997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:36:24.951179 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.951139    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"]
Jul 07 01:36:24.951406 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.951368    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv" podUID="bc1eb203-3029-4e64-92b9-808b32c49ac3" containerName="maas-api" containerID="cri-o://2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a" gracePeriod=30
Jul 07 01:36:24.974782 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:24.974752    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:36:25.407072 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.407043    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:36:25.573198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.573090    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn6rp\" (UniqueName: \"kubernetes.io/projected/bc1eb203-3029-4e64-92b9-808b32c49ac3-kube-api-access-mn6rp\") pod \"bc1eb203-3029-4e64-92b9-808b32c49ac3\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") "
Jul 07 01:36:25.573198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.573152    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1eb203-3029-4e64-92b9-808b32c49ac3-openshift-service-ca\") pod \"bc1eb203-3029-4e64-92b9-808b32c49ac3\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") "
Jul 07 01:36:25.573198 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.573180    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls\") pod \"bc1eb203-3029-4e64-92b9-808b32c49ac3\" (UID: \"bc1eb203-3029-4e64-92b9-808b32c49ac3\") "
Jul 07 01:36:25.573563 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.573533    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1eb203-3029-4e64-92b9-808b32c49ac3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc1eb203-3029-4e64-92b9-808b32c49ac3" (UID: "bc1eb203-3029-4e64-92b9-808b32c49ac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:36:25.575551 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.575505    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc1eb203-3029-4e64-92b9-808b32c49ac3" (UID: "bc1eb203-3029-4e64-92b9-808b32c49ac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:36:25.575551 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.575536    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1eb203-3029-4e64-92b9-808b32c49ac3-kube-api-access-mn6rp" (OuterVolumeSpecName: "kube-api-access-mn6rp") pod "bc1eb203-3029-4e64-92b9-808b32c49ac3" (UID: "bc1eb203-3029-4e64-92b9-808b32c49ac3"). InnerVolumeSpecName "kube-api-access-mn6rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:36:25.674398 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.674352    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn6rp\" (UniqueName: \"kubernetes.io/projected/bc1eb203-3029-4e64-92b9-808b32c49ac3-kube-api-access-mn6rp\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:25.674398 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.674390    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1eb203-3029-4e64-92b9-808b32c49ac3-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:25.674398 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:25.674400    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1eb203-3029-4e64-92b9-808b32c49ac3-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:26.057565 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.057529    2562 generic.go:356] "Generic (PLEG): container finished" podID="bc1eb203-3029-4e64-92b9-808b32c49ac3" containerID="2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a" exitCode=0
Jul 07 01:36:26.057805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.057604    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"
Jul 07 01:36:26.057805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.057614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv" event={"ID":"bc1eb203-3029-4e64-92b9-808b32c49ac3","Type":"ContainerDied","Data":"2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a"}
Jul 07 01:36:26.057805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.057714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv" event={"ID":"bc1eb203-3029-4e64-92b9-808b32c49ac3","Type":"ContainerDied","Data":"92bad507665e34817cb333cfce69e75c926f6453c501884ddb454acf41ef4e95"}
Jul 07 01:36:26.057805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.057738    2562 scope.go:117] "RemoveContainer" containerID="2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a"
Jul 07 01:36:26.067455 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.067434    2562 scope.go:117] "RemoveContainer" containerID="2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a"
Jul 07 01:36:26.067790 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:36:26.067769    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a\": container with ID starting with 2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a not found: ID does not exist" containerID="2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a"
Jul 07 01:36:26.067874 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.067800    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a"} err="failed to get container status \"2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a\": rpc error: code = NotFound desc = could not find container \"2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a\": container with ID starting with 2aafed7194d68d523cd930546aa34b91333d7f0baba093eda376315cc9c1d41a not found: ID does not exist"
Jul 07 01:36:26.079084 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.079056    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"]
Jul 07 01:36:26.082377 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.082349    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-86cc19-5758fc7947-5svcv"]
Jul 07 01:36:26.291456 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:26.291416    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1eb203-3029-4e64-92b9-808b32c49ac3" path="/var/lib/kubelet/pods/bc1eb203-3029-4e64-92b9-808b32c49ac3/volumes"
Jul 07 01:36:29.847112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:29.847075    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:36:29.851719 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:29.851669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:36:38.306328 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.306280    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8" podUID="6bcabd39-11d8-4890-a211-1632c8ca5d1a" containerName="main" containerID="cri-o://6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336" gracePeriod=45
Jul 07 01:36:38.574858 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.574829    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:36:38.587086 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587049    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-dshm\") pod \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") "
Jul 07 01:36:38.587284 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587121    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tmp-dir\") pod \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") "
Jul 07 01:36:38.587284 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdhcd\" (UniqueName: \"kubernetes.io/projected/6bcabd39-11d8-4890-a211-1632c8ca5d1a-kube-api-access-vdhcd\") pod \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") "
Jul 07 01:36:38.587284 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587189    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-home\") pod \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") "
Jul 07 01:36:38.587284 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587228    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-model-cache\") pod \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") "
Jul 07 01:36:38.587284 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587255    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tls-certs\") pod \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\" (UID: \"6bcabd39-11d8-4890-a211-1632c8ca5d1a\") "
Jul 07 01:36:38.587555 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587378    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bcabd39-11d8-4890-a211-1632c8ca5d1a" (UID: "6bcabd39-11d8-4890-a211-1632c8ca5d1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:38.587555 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587456    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-home" (OuterVolumeSpecName: "home") pod "6bcabd39-11d8-4890-a211-1632c8ca5d1a" (UID: "6bcabd39-11d8-4890-a211-1632c8ca5d1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:38.587663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587563    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:38.587663 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587608    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:38.587816 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.587710    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bcabd39-11d8-4890-a211-1632c8ca5d1a" (UID: "6bcabd39-11d8-4890-a211-1632c8ca5d1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:38.589706 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.589654    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-dshm" (OuterVolumeSpecName: "dshm") pod "6bcabd39-11d8-4890-a211-1632c8ca5d1a" (UID: "6bcabd39-11d8-4890-a211-1632c8ca5d1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:38.590045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.589996    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcabd39-11d8-4890-a211-1632c8ca5d1a-kube-api-access-vdhcd" (OuterVolumeSpecName: "kube-api-access-vdhcd") pod "6bcabd39-11d8-4890-a211-1632c8ca5d1a" (UID: "6bcabd39-11d8-4890-a211-1632c8ca5d1a"). InnerVolumeSpecName "kube-api-access-vdhcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:36:38.590045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.590005    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bcabd39-11d8-4890-a211-1632c8ca5d1a" (UID: "6bcabd39-11d8-4890-a211-1632c8ca5d1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:36:38.688463 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.688417    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhcd\" (UniqueName: \"kubernetes.io/projected/6bcabd39-11d8-4890-a211-1632c8ca5d1a-kube-api-access-vdhcd\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:38.688463 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.688457    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:38.688463 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.688467    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bcabd39-11d8-4890-a211-1632c8ca5d1a-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:38.688463 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:38.688475    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bcabd39-11d8-4890-a211-1632c8ca5d1a-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:39.122361 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.122321    2562 generic.go:356] "Generic (PLEG): container finished" podID="6bcabd39-11d8-4890-a211-1632c8ca5d1a" containerID="6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336" exitCode=0
Jul 07 01:36:39.122361 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.122368    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"
Jul 07 01:36:39.122607 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.122404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8" event={"ID":"6bcabd39-11d8-4890-a211-1632c8ca5d1a","Type":"ContainerDied","Data":"6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336"}
Jul 07 01:36:39.122607 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.122450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8" event={"ID":"6bcabd39-11d8-4890-a211-1632c8ca5d1a","Type":"ContainerDied","Data":"d89326b096e374c9462f0360f1d7ca9e072ce686442f023581996a4dec1a102b"}
Jul 07 01:36:39.122607 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.122478    2562 scope.go:117] "RemoveContainer" containerID="6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336"
Jul 07 01:36:39.132320 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.132295    2562 scope.go:117] "RemoveContainer" containerID="6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336"
Jul 07 01:36:39.132775 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:36:39.132754    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336\": container with ID starting with 6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336 not found: ID does not exist" containerID="6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336"
Jul 07 01:36:39.132868 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.132784    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336"} err="failed to get container status \"6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336\": rpc error: code = NotFound desc = could not find container \"6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336\": container with ID starting with 6bc97ac6ca2af74432bc488d13e9e9b888dfef6716c7245f1826a932fc40f336 not found: ID does not exist"
Jul 07 01:36:39.143367 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.143328    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"]
Jul 07 01:36:39.149358 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.149325    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-86cc19/test-model-86cc19-kserve-7477486bf7-x7nq8"]
Jul 07 01:36:39.183946 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.183903    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg" podUID="2c823efb-f178-4e7f-902f-f29511affa0f" containerName="main" containerID="cri-o://25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937" gracePeriod=45
Jul 07 01:36:39.453392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.453357    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:36:39.495212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495169    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-tmp-dir\") pod \"2c823efb-f178-4e7f-902f-f29511affa0f\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") "
Jul 07 01:36:39.495426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495227    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-model-cache\") pod \"2c823efb-f178-4e7f-902f-f29511affa0f\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") "
Jul 07 01:36:39.495426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c823efb-f178-4e7f-902f-f29511affa0f-tls-certs\") pod \"2c823efb-f178-4e7f-902f-f29511affa0f\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") "
Jul 07 01:36:39.495426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495302    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-home\") pod \"2c823efb-f178-4e7f-902f-f29511affa0f\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") "
Jul 07 01:36:39.495426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htlq2\" (UniqueName: \"kubernetes.io/projected/2c823efb-f178-4e7f-902f-f29511affa0f-kube-api-access-htlq2\") pod \"2c823efb-f178-4e7f-902f-f29511affa0f\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") "
Jul 07 01:36:39.495426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495421    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-dshm\") pod \"2c823efb-f178-4e7f-902f-f29511affa0f\" (UID: \"2c823efb-f178-4e7f-902f-f29511affa0f\") "
Jul 07 01:36:39.495728 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495466    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c823efb-f178-4e7f-902f-f29511affa0f" (UID: "2c823efb-f178-4e7f-902f-f29511affa0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:39.495728 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495494    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c823efb-f178-4e7f-902f-f29511affa0f" (UID: "2c823efb-f178-4e7f-902f-f29511affa0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:39.495728 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495577    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-home" (OuterVolumeSpecName: "home") pod "2c823efb-f178-4e7f-902f-f29511affa0f" (UID: "2c823efb-f178-4e7f-902f-f29511affa0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:39.495897 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495750    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:39.495897 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495765    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:39.495897 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.495773    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:39.497983 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.497959    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c823efb-f178-4e7f-902f-f29511affa0f-kube-api-access-htlq2" (OuterVolumeSpecName: "kube-api-access-htlq2") pod "2c823efb-f178-4e7f-902f-f29511affa0f" (UID: "2c823efb-f178-4e7f-902f-f29511affa0f"). InnerVolumeSpecName "kube-api-access-htlq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:36:39.497983 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.497965    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c823efb-f178-4e7f-902f-f29511affa0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c823efb-f178-4e7f-902f-f29511affa0f" (UID: "2c823efb-f178-4e7f-902f-f29511affa0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:36:39.498107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.498011    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-dshm" (OuterVolumeSpecName: "dshm") pod "2c823efb-f178-4e7f-902f-f29511affa0f" (UID: "2c823efb-f178-4e7f-902f-f29511affa0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:36:39.596987 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.596943    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlq2\" (UniqueName: \"kubernetes.io/projected/2c823efb-f178-4e7f-902f-f29511affa0f-kube-api-access-htlq2\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:39.596987 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.596979    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c823efb-f178-4e7f-902f-f29511affa0f-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:39.596987 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:39.596989    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c823efb-f178-4e7f-902f-f29511affa0f-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:36:40.129939 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.129899    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c823efb-f178-4e7f-902f-f29511affa0f" containerID="25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937" exitCode=0
Jul 07 01:36:40.130128 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.129946    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"
Jul 07 01:36:40.130128 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.129956    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg" event={"ID":"2c823efb-f178-4e7f-902f-f29511affa0f","Type":"ContainerDied","Data":"25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937"}
Jul 07 01:36:40.130128 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.129996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg" event={"ID":"2c823efb-f178-4e7f-902f-f29511affa0f","Type":"ContainerDied","Data":"49c0713c99f26a2e77e6fea0b534ac215dfc5532a02843578677efa633567b3d"}
Jul 07 01:36:40.130128 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.130018    2562 scope.go:117] "RemoveContainer" containerID="25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937"
Jul 07 01:36:40.139237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.139219    2562 scope.go:117] "RemoveContainer" containerID="25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937"
Jul 07 01:36:40.139543 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:36:40.139521    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937\": container with ID starting with 25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937 not found: ID does not exist" containerID="25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937"
Jul 07 01:36:40.139641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.139548    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937"} err="failed to get container status \"25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937\": rpc error: code = NotFound desc = could not find container \"25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937\": container with ID starting with 25f23e334d4a26f403e32cdd10b67939189db22bdb0c8654e1b52ed16cb9e937 not found: ID does not exist"
Jul 07 01:36:40.153119 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.153088    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"]
Jul 07 01:36:40.156245 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.156216    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cf6840/test-model-cf6840-kserve-f45b9fb8c-96mbg"]
Jul 07 01:36:40.290338 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.290295    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c823efb-f178-4e7f-902f-f29511affa0f" path="/var/lib/kubelet/pods/2c823efb-f178-4e7f-902f-f29511affa0f/volumes"
Jul 07 01:36:40.290815 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:36:40.290795    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bcabd39-11d8-4890-a211-1632c8ca5d1a" path="/var/lib/kubelet/pods/6bcabd39-11d8-4890-a211-1632c8ca5d1a/volumes"
Jul 07 01:37:06.598565 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:06.598439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:06.845015 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:06.844977    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"]
Jul 07 01:37:06.845277 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:06.845243    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy" containerID="cri-o://9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990" gracePeriod=30
Jul 07 01:37:06.870132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:06.870024    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:07.098615 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:07.098568    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:07.245431 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:07.245395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:07.648175 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:07.648081    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:07.748952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:07.748917    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:07.879605 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:07.879566    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"]
Jul 07 01:37:07.879846 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:07.879819    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp" podUID="66149d33-4040-4980-91c8-39d25c0d9e46" containerName="maas-api" containerID="cri-o://b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470" gracePeriod=30
Jul 07 01:37:07.901189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:07.901114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:08.139710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.139663    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:37:08.251383 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.251346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh5nw\" (UniqueName: \"kubernetes.io/projected/66149d33-4040-4980-91c8-39d25c0d9e46-kube-api-access-nh5nw\") pod \"66149d33-4040-4980-91c8-39d25c0d9e46\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") "
Jul 07 01:37:08.251595 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.251405    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls\") pod \"66149d33-4040-4980-91c8-39d25c0d9e46\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") "
Jul 07 01:37:08.251595 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.251477    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66149d33-4040-4980-91c8-39d25c0d9e46-openshift-service-ca\") pod \"66149d33-4040-4980-91c8-39d25c0d9e46\" (UID: \"66149d33-4040-4980-91c8-39d25c0d9e46\") "
Jul 07 01:37:08.251839 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.251812    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66149d33-4040-4980-91c8-39d25c0d9e46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66149d33-4040-4980-91c8-39d25c0d9e46" (UID: "66149d33-4040-4980-91c8-39d25c0d9e46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:37:08.253760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.253734    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66149d33-4040-4980-91c8-39d25c0d9e46-kube-api-access-nh5nw" (OuterVolumeSpecName: "kube-api-access-nh5nw") pod "66149d33-4040-4980-91c8-39d25c0d9e46" (UID: "66149d33-4040-4980-91c8-39d25c0d9e46"). InnerVolumeSpecName "kube-api-access-nh5nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:37:08.253853 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.253803    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66149d33-4040-4980-91c8-39d25c0d9e46" (UID: "66149d33-4040-4980-91c8-39d25c0d9e46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:37:08.268537 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.268507    2562 generic.go:356] "Generic (PLEG): container finished" podID="66149d33-4040-4980-91c8-39d25c0d9e46" containerID="b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470" exitCode=0
Jul 07 01:37:08.268745 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.268558    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"
Jul 07 01:37:08.268745 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.268594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp" event={"ID":"66149d33-4040-4980-91c8-39d25c0d9e46","Type":"ContainerDied","Data":"b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470"}
Jul 07 01:37:08.268745 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.268660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp" event={"ID":"66149d33-4040-4980-91c8-39d25c0d9e46","Type":"ContainerDied","Data":"7b6b8bf89df32cbbe2d12adcb3dce3a38098610e9239e2d9eb4fdd6095033e8b"}
Jul 07 01:37:08.268745 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.268718    2562 scope.go:117] "RemoveContainer" containerID="b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470"
Jul 07 01:37:08.277974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.277954    2562 scope.go:117] "RemoveContainer" containerID="b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470"
Jul 07 01:37:08.278293 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:08.278272    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470\": container with ID starting with b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470 not found: ID does not exist" containerID="b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470"
Jul 07 01:37:08.278340 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.278304    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470"} err="failed to get container status \"b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470\": rpc error: code = NotFound desc = could not find container \"b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470\": container with ID starting with b18913810b7b4ba4c445691a10c13f52cd01348532d816d4bcf86414f84ab470 not found: ID does not exist"
Jul 07 01:37:08.289353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.289320    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"]
Jul 07 01:37:08.295258 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.295226    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cf6840-7c8455589c-c45tp"]
Jul 07 01:37:08.352819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.352782    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh5nw\" (UniqueName: \"kubernetes.io/projected/66149d33-4040-4980-91c8-39d25c0d9e46-kube-api-access-nh5nw\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:08.352819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.352814    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66149d33-4040-4980-91c8-39d25c0d9e46-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:08.352819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.352826    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66149d33-4040-4980-91c8-39d25c0d9e46-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:08.480966 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.480928    2562 patch_prober.go:28] interesting pod/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 01:37:08.481149 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:08.481003    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 01:37:10.289642 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:10.289596    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66149d33-4040-4980-91c8-39d25c0d9e46" path="/var/lib/kubelet/pods/66149d33-4040-4980-91c8-39d25c0d9e46/volumes"
Jul 07 01:37:10.740237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:10.740197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:37:12.104487 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.104459    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:37:12.189434 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189337    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96cjf\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-kube-api-access-96cjf\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189434 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189381    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-data\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189434 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189402    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-certs\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189512    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-podinfo\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189562    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istiod-ca-cert\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189648    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-envoy\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189671    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:12.189760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189698    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-socket\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189731    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-credential-socket\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.189760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189756    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-token\") pod \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\" (UID: \"04d88ac5-a6c5-4edd-8095-3d2e17f93507\") "
Jul 07 01:37:12.190125 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189943    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:12.190125 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.189974    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:12.190125 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.190005    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:37:12.190125 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.190051    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.190125 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.190067    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.190558 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.190354    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-data" (OuterVolumeSpecName: "istio-data") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:12.192026 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.191999    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:37:12.192026 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.192015    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-kube-api-access-96cjf" (OuterVolumeSpecName: "kube-api-access-96cjf") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "kube-api-access-96cjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:37:12.192187 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.192030    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:12.192233 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.192218    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-token" (OuterVolumeSpecName: "istio-token") pod "04d88ac5-a6c5-4edd-8095-3d2e17f93507" (UID: "04d88ac5-a6c5-4edd-8095-3d2e17f93507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:37:12.290502 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.290459    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.290502 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.290499    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.290502 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.290509    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.290779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.290519    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96cjf\" (UniqueName: \"kubernetes.io/projected/04d88ac5-a6c5-4edd-8095-3d2e17f93507-kube-api-access-96cjf\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.290779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.290528    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.290779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.290536    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.290779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.290543    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d88ac5-a6c5-4edd-8095-3d2e17f93507-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:12.292199 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.292167    2562 generic.go:356] "Generic (PLEG): container finished" podID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerID="9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990" exitCode=0
Jul 07 01:37:12.292335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.292226    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"
Jul 07 01:37:12.292335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.292246    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" event={"ID":"04d88ac5-a6c5-4edd-8095-3d2e17f93507","Type":"ContainerDied","Data":"9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990"}
Jul 07 01:37:12.292335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.292307    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6" event={"ID":"04d88ac5-a6c5-4edd-8095-3d2e17f93507","Type":"ContainerDied","Data":"1ad21fe45107691d77f600d7990cb16a383ea50f845310d20e1d68a4b820c173"}
Jul 07 01:37:12.292509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.292338    2562 scope.go:117] "RemoveContainer" containerID="9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990"
Jul 07 01:37:12.304237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.304215    2562 scope.go:117] "RemoveContainer" containerID="cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc"
Jul 07 01:37:12.313210 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.313189    2562 scope.go:117] "RemoveContainer" containerID="9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990"
Jul 07 01:37:12.313514 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:12.313497    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990\": container with ID starting with 9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990 not found: ID does not exist" containerID="9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990"
Jul 07 01:37:12.313568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.313522    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990"} err="failed to get container status \"9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990\": rpc error: code = NotFound desc = could not find container \"9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990\": container with ID starting with 9fdcaf8ac2ff7535ec3ef2fded27fad44bf26c0f98a9ffec59c6e2dac9b63990 not found: ID does not exist"
Jul 07 01:37:12.313568 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.313541    2562 scope.go:117] "RemoveContainer" containerID="cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc"
Jul 07 01:37:12.313816 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:12.313796    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc\": container with ID starting with cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc not found: ID does not exist" containerID="cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc"
Jul 07 01:37:12.313863 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.313823    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc"} err="failed to get container status \"cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc\": rpc error: code = NotFound desc = could not find container \"cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc\": container with ID starting with cd9c85db2af0dba8cb98b66bd45a64d4b2b9f63e838668b4776d712161d05edc not found: ID does not exist"
Jul 07 01:37:12.316103 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.316081    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"]
Jul 07 01:37:12.322636 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:12.322610    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-86cc19-openshift-default-557dd5b589-6hxc6"]
Jul 07 01:37:13.342151 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:13.342118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:13.404964 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:13.404924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:13.621657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:13.621583    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:13.837972 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:13.837937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:14.166099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.166062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:14.289287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.289251    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" path="/var/lib/kubelet/pods/04d88ac5-a6c5-4edd-8095-3d2e17f93507/volumes"
Jul 07 01:37:14.441358 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441271    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"]
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441596    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441607    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441620    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441625    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441633    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c823efb-f178-4e7f-902f-f29511affa0f" containerName="main"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441638    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c823efb-f178-4e7f-902f-f29511affa0f" containerName="main"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441653    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66149d33-4040-4980-91c8-39d25c0d9e46" containerName="maas-api"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441659    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="66149d33-4040-4980-91c8-39d25c0d9e46" containerName="maas-api"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441664    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcabd39-11d8-4890-a211-1632c8ca5d1a" containerName="main"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441716    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcabd39-11d8-4890-a211-1632c8ca5d1a" containerName="main"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441723    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1eb203-3029-4e64-92b9-808b32c49ac3" containerName="maas-api"
Jul 07 01:37:14.441771 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441728    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1eb203-3029-4e64-92b9-808b32c49ac3" containerName="maas-api"
Jul 07 01:37:14.442202 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441789    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy"
Jul 07 01:37:14.442202 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441797    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d88ac5-a6c5-4edd-8095-3d2e17f93507" containerName="istio-proxy"
Jul 07 01:37:14.442202 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441803    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c823efb-f178-4e7f-902f-f29511affa0f" containerName="main"
Jul 07 01:37:14.442202 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441808    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="66149d33-4040-4980-91c8-39d25c0d9e46" containerName="maas-api"
Jul 07 01:37:14.442202 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441815    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1eb203-3029-4e64-92b9-808b32c49ac3" containerName="maas-api"
Jul 07 01:37:14.442202 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.441820    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcabd39-11d8-4890-a211-1632c8ca5d1a" containerName="main"
Jul 07 01:37:14.449300 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.448284    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.451562 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.451523    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:37:14.451737 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.451607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9cd09e-openshift-default-dockercfg-mn6cs\""
Jul 07 01:37:14.457647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.457624    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"]
Jul 07 01:37:14.464964 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.464938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:14.611487 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-certs\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611487 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtlg\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-kube-api-access-hxtlg\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-credential-socket\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-envoy\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-data\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611658    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istiod-ca-cert\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-podinfo\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611737    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-token\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.611819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.611809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-socket\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-podinfo\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-token\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713528 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-socket\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713528 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-certs\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713528 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtlg\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-kube-api-access-hxtlg\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713528 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-credential-socket\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713528 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-envoy\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713905 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-data\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713905 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istiod-ca-cert\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713905 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713833    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-socket\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.713905 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713894    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-certs\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.714274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.713932    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-credential-socket\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.714274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.714197    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-data\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.714274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.714262    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istiod-ca-cert\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.716069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.716043    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-podinfo\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.716176 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.716134    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-envoy\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.721401 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.721370    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-token\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.721592 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.721570    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtlg\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-kube-api-access-hxtlg\") pod \"e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") " pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.762562 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.762523    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:14.825842 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.825802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:14.908168 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.908142    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"]
Jul 07 01:37:14.910868 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:37:14.910844    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39d1bfe_fe2d_4c16_9595_8020ac2687b0.slice/crio-b59c058032a64f2c3f3325625d6d6280d0f422f2c08b340170c657bc27b44609 WatchSource:0}: Error finding container b59c058032a64f2c3f3325625d6d6280d0f422f2c08b340170c657bc27b44609: Status 404 returned error can't find the container with id b59c058032a64f2c3f3325625d6d6280d0f422f2c08b340170c657bc27b44609
Jul 07 01:37:14.913158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.913123    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:14.913247 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.913200    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:14.913247 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:14.913231    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:15.309239 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:15.309197    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" event={"ID":"e39d1bfe-fe2d-4c16-9595-8020ac2687b0","Type":"ContainerStarted","Data":"d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb"}
Jul 07 01:37:15.309435 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:15.309253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" event={"ID":"e39d1bfe-fe2d-4c16-9595-8020ac2687b0","Type":"ContainerStarted","Data":"b59c058032a64f2c3f3325625d6d6280d0f422f2c08b340170c657bc27b44609"}
Jul 07 01:37:15.329218 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:15.329171    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" podStartSLOduration=1.329153533 podStartE2EDuration="1.329153533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:37:15.327650978 +0000 UTC m=+2163.602642304" watchObservedRunningTime="2026-07-07 01:37:15.329153533 +0000 UTC m=+2163.604144854"
Jul 07 01:37:15.763192 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:15.763149    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:15.768047 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:15.768019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:16.314157 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:16.314128    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:16.315230 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:16.315208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:17.225294 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.225256    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:17.300976 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.300941    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"]
Jul 07 01:37:17.304844 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.304824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.307246 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.307219    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jtxhz\""
Jul 07 01:37:17.307396 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.307248    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-9cd09e-serving-cert\""
Jul 07 01:37:17.316612 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.316563    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"]
Jul 07 01:37:17.318531 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.318494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:17.379112 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.379080    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:17.440061 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.440019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4vr\" (UniqueName: \"kubernetes.io/projected/e5224ffc-5d41-4434-9bb0-75c486c41162-kube-api-access-gd4vr\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.440275 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.440131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.440275 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.440168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5224ffc-5d41-4434-9bb0-75c486c41162-openshift-service-ca\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.541099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.540995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.541099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.541052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5224ffc-5d41-4434-9bb0-75c486c41162-openshift-service-ca\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.541308 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.541115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4vr\" (UniqueName: \"kubernetes.io/projected/e5224ffc-5d41-4434-9bb0-75c486c41162-kube-api-access-gd4vr\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.541308 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:17.541168    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-9cd09e-serving-cert: secret "maas-api-e2e-api-a-9cd09e-serving-cert" not found
Jul 07 01:37:17.541308 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:17.541237    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls podName:e5224ffc-5d41-4434-9bb0-75c486c41162 nodeName:}" failed. No retries permitted until 2026-07-07 01:37:18.041215381 +0000 UTC m=+2166.316206685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls") pod "maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x" (UID: "e5224ffc-5d41-4434-9bb0-75c486c41162") : secret "maas-api-e2e-api-a-9cd09e-serving-cert" not found
Jul 07 01:37:17.541732 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.541714    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5224ffc-5d41-4434-9bb0-75c486c41162-openshift-service-ca\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.553812 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.553784    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4vr\" (UniqueName: \"kubernetes.io/projected/e5224ffc-5d41-4434-9bb0-75c486c41162-kube-api-access-gd4vr\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:17.659552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.659517    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:17.779210 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.779173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:17.880126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:17.880038    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:18.046826 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:18.046782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:18.049393 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:18.049369    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls\") pod \"maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") " pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:18.217152 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:18.217107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:18.367488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:18.367454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"]
Jul 07 01:37:18.374112 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:37:18.374076    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5224ffc_5d41_4434_9bb0_75c486c41162.slice/crio-3c92696e92884b74042be57d1531701b352df288c31d7d2c85f44fc48dc03d90 WatchSource:0}: Error finding container 3c92696e92884b74042be57d1531701b352df288c31d7d2c85f44fc48dc03d90: Status 404 returned error can't find the container with id 3c92696e92884b74042be57d1531701b352df288c31d7d2c85f44fc48dc03d90
Jul 07 01:37:19.331866 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:19.331821    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x" event={"ID":"e5224ffc-5d41-4434-9bb0-75c486c41162","Type":"ContainerStarted","Data":"f1ec4aa6210ac9cdcde189b8987edab93b732fcea7548e2a78406b78ab9aab60"}
Jul 07 01:37:19.332053 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:19.331894    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:19.332053 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:19.331917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x" event={"ID":"e5224ffc-5d41-4434-9bb0-75c486c41162","Type":"ContainerStarted","Data":"3c92696e92884b74042be57d1531701b352df288c31d7d2c85f44fc48dc03d90"}
Jul 07 01:37:19.348502 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:19.348451    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x" podStartSLOduration=1.9424409809999998 podStartE2EDuration="2.348435133s" totalImagesPullingTime="405.994152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:37:19.346211432 +0000 UTC m=+2167.621202756" watchObservedRunningTime="2026-07-07 01:37:19.348435133 +0000 UTC m=+2167.623426446"
Jul 07 01:37:21.698655 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:21.698614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:22.147245 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:22.147198    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:22.450940 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:22.450848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:22.578910 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:22.578880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:24.237005 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:24.236963    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:37:25.149742 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.149707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:25.239878 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.239676    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:25.247827 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.245059    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"]
Jul 07 01:37:25.249642 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.249614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.252297 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.252272    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-df671c-serving-cert\""
Jul 07 01:37:25.259259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.259231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"]
Jul 07 01:37:25.300655 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.300613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.300655 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.300655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a0361-9b33-4e9c-8566-ecde09242d9e-openshift-service-ca\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.300937 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.300702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwjb\" (UniqueName: \"kubernetes.io/projected/ec1a0361-9b33-4e9c-8566-ecde09242d9e-kube-api-access-cfwjb\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.307240 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.307210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:25.352036 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.352006    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:25.401631 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.401529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.401631 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.401570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a0361-9b33-4e9c-8566-ecde09242d9e-openshift-service-ca\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.401631 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.401603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwjb\" (UniqueName: \"kubernetes.io/projected/ec1a0361-9b33-4e9c-8566-ecde09242d9e-kube-api-access-cfwjb\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.401949 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:25.401732    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-df671c-serving-cert: secret "maas-api-e2e-api-b-df671c-serving-cert" not found
Jul 07 01:37:25.401949 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:25.401816    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls podName:ec1a0361-9b33-4e9c-8566-ecde09242d9e nodeName:}" failed. No retries permitted until 2026-07-07 01:37:25.901793702 +0000 UTC m=+2174.176785008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls") pod "maas-api-e2e-api-b-df671c-85858bd6d-84wq8" (UID: "ec1a0361-9b33-4e9c-8566-ecde09242d9e") : secret "maas-api-e2e-api-b-df671c-serving-cert" not found
Jul 07 01:37:25.402324 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.402301    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a0361-9b33-4e9c-8566-ecde09242d9e-openshift-service-ca\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.414371 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.414340    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwjb\" (UniqueName: \"kubernetes.io/projected/ec1a0361-9b33-4e9c-8566-ecde09242d9e-kube-api-access-cfwjb\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.802559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.802520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:25.902885 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.902850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:25.907144 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.907120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:25.909792 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:25.909766    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls\") pod \"maas-api-e2e-api-b-df671c-85858bd6d-84wq8\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") " pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:26.005884 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:26.005850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:26.164326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:26.164220    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:26.367451 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:26.367374    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"]
Jul 07 01:37:26.370222 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:37:26.370178    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1a0361_9b33_4e9c_8566_ecde09242d9e.slice/crio-81e46a3d86d05f45671d7db5482dffc8b9aa7ca519c3781586e2ad3b114c9c55 WatchSource:0}: Error finding container 81e46a3d86d05f45671d7db5482dffc8b9aa7ca519c3781586e2ad3b114c9c55: Status 404 returned error can't find the container with id 81e46a3d86d05f45671d7db5482dffc8b9aa7ca519c3781586e2ad3b114c9c55
Jul 07 01:37:27.376662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:27.376626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8" event={"ID":"ec1a0361-9b33-4e9c-8566-ecde09242d9e","Type":"ContainerStarted","Data":"e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2"}
Jul 07 01:37:27.377089 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:27.376694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8" event={"ID":"ec1a0361-9b33-4e9c-8566-ecde09242d9e","Type":"ContainerStarted","Data":"81e46a3d86d05f45671d7db5482dffc8b9aa7ca519c3781586e2ad3b114c9c55"}
Jul 07 01:37:27.377089 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:27.376758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:27.393546 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:27.393488    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8" podStartSLOduration=1.973385907 podStartE2EDuration="2.393470711s" totalImagesPullingTime="420.084804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:37:27.392002161 +0000 UTC m=+2175.666993485" watchObservedRunningTime="2026-07-07 01:37:27.393470711 +0000 UTC m=+2175.668462032"
Jul 07 01:37:29.623586 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:29.623516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:31.641002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:31.640058    2562 patch_prober.go:28] interesting pod/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:52860->10.133.0.51:15021: read: connection reset by peer" start-of-body=
Jul 07 01:37:31.641002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:31.640127    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:52860->10.133.0.51:15021: read: connection reset by peer"
Jul 07 01:37:32.407570 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:32.407535    2562 generic.go:356] "Generic (PLEG): container finished" podID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerID="d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb" exitCode=0
Jul 07 01:37:32.407791 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:32.407611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" event={"ID":"e39d1bfe-fe2d-4c16-9595-8020ac2687b0","Type":"ContainerDied","Data":"d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb"}
Jul 07 01:37:32.408047 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:32.408032    2562 scope.go:117] "RemoveContainer" containerID="d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb"
Jul 07 01:37:32.409045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:32.409017    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:32.409111 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:32.409089    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:32.409146 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:32.409119    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:33.390197 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:33.390169    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:33.414765 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:33.414726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" event={"ID":"e39d1bfe-fe2d-4c16-9595-8020ac2687b0","Type":"ContainerStarted","Data":"a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1"}
Jul 07 01:37:33.763578 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:33.763533    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:33.763875 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:33.763853    2562 patch_prober.go:28] interesting pod/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 01:37:33.763972 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:33.763906    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 01:37:34.763844 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:34.763800    2562 patch_prober.go:28] interesting pod/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 01:37:34.764278 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:34.763874    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 01:37:35.757104 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:35.757066    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:37:35.763941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:35.763907    2562 patch_prober.go:28] interesting pod/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 01:37:35.764328 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:35.763967    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 01:37:36.763863 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:36.763820    2562 patch_prober.go:28] interesting pod/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 01:37:36.764099 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:36.763892    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 01:37:37.592092 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:37.592051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:37.767058 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:37.767019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:37.767516 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:37.767303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:37.768152 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:37.768134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:37.965742 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:37.965699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:38.068872 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.068832    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:38.166798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.166747    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:38.241422 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.241328    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"]
Jul 07 01:37:38.241601 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.241567    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x" podUID="e5224ffc-5d41-4434-9bb0-75c486c41162" containerName="maas-api" containerID="cri-o://f1ec4aa6210ac9cdcde189b8987edab93b732fcea7548e2a78406b78ab9aab60" gracePeriod=30
Jul 07 01:37:38.270800 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.270770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:38.451618 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.451587    2562 generic.go:356] "Generic (PLEG): container finished" podID="e5224ffc-5d41-4434-9bb0-75c486c41162" containerID="f1ec4aa6210ac9cdcde189b8987edab93b732fcea7548e2a78406b78ab9aab60" exitCode=0
Jul 07 01:37:38.451826 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.451671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x" event={"ID":"e5224ffc-5d41-4434-9bb0-75c486c41162","Type":"ContainerDied","Data":"f1ec4aa6210ac9cdcde189b8987edab93b732fcea7548e2a78406b78ab9aab60"}
Jul 07 01:37:38.509259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.509227    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:38.628992 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.628960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd4vr\" (UniqueName: \"kubernetes.io/projected/e5224ffc-5d41-4434-9bb0-75c486c41162-kube-api-access-gd4vr\") pod \"e5224ffc-5d41-4434-9bb0-75c486c41162\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") "
Jul 07 01:37:38.629218 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.629022    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5224ffc-5d41-4434-9bb0-75c486c41162-openshift-service-ca\") pod \"e5224ffc-5d41-4434-9bb0-75c486c41162\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") "
Jul 07 01:37:38.629218 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.629048    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls\") pod \"e5224ffc-5d41-4434-9bb0-75c486c41162\" (UID: \"e5224ffc-5d41-4434-9bb0-75c486c41162\") "
Jul 07 01:37:38.629420 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.629393    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5224ffc-5d41-4434-9bb0-75c486c41162-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5224ffc-5d41-4434-9bb0-75c486c41162" (UID: "e5224ffc-5d41-4434-9bb0-75c486c41162"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:37:38.631399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.631375    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5224ffc-5d41-4434-9bb0-75c486c41162-kube-api-access-gd4vr" (OuterVolumeSpecName: "kube-api-access-gd4vr") pod "e5224ffc-5d41-4434-9bb0-75c486c41162" (UID: "e5224ffc-5d41-4434-9bb0-75c486c41162"). InnerVolumeSpecName "kube-api-access-gd4vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:37:38.631480 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.631415    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5224ffc-5d41-4434-9bb0-75c486c41162" (UID: "e5224ffc-5d41-4434-9bb0-75c486c41162"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:37:38.730104 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.730058    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5224ffc-5d41-4434-9bb0-75c486c41162-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:38.730104 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.730091    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5224ffc-5d41-4434-9bb0-75c486c41162-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:38.730104 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:38.730100    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4vr\" (UniqueName: \"kubernetes.io/projected/e5224ffc-5d41-4434-9bb0-75c486c41162-kube-api-access-gd4vr\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:39.458091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:39.458055    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"
Jul 07 01:37:39.458091 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:39.458083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x" event={"ID":"e5224ffc-5d41-4434-9bb0-75c486c41162","Type":"ContainerDied","Data":"3c92696e92884b74042be57d1531701b352df288c31d7d2c85f44fc48dc03d90"}
Jul 07 01:37:39.458597 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:39.458145    2562 scope.go:117] "RemoveContainer" containerID="f1ec4aa6210ac9cdcde189b8987edab93b732fcea7548e2a78406b78ab9aab60"
Jul 07 01:37:39.480063 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:39.480032    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"]
Jul 07 01:37:39.481734 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:39.481707    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-9cd09e-779b8cf57b-8272x"]
Jul 07 01:37:40.295755 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:40.295714    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5224ffc-5d41-4434-9bb0-75c486c41162" path="/var/lib/kubelet/pods/e5224ffc-5d41-4434-9bb0-75c486c41162/volumes"
Jul 07 01:37:43.703027 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:43.702997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:43.801746 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:43.801667    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:43.931594 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:43.931556    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"]
Jul 07 01:37:43.931896 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:43.931857    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy" containerID="cri-o://a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1" gracePeriod=30
Jul 07 01:37:43.954701 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:43.954599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:44.168189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:44.168156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:44.351240 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:44.351206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:44.797307 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:44.797267    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:44.912830 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:44.912790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:45.014046 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.013972    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:45.146924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.146823    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"]
Jul 07 01:37:45.147446 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.147400    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8" podUID="ec1a0361-9b33-4e9c-8566-ecde09242d9e" containerName="maas-api" containerID="cri-o://e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2" gracePeriod=30
Jul 07 01:37:45.171531 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.171498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:45.409555 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.409484    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:45.488961 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.488933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls\") pod \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") "
Jul 07 01:37:45.489153 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.489019    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a0361-9b33-4e9c-8566-ecde09242d9e-openshift-service-ca\") pod \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") "
Jul 07 01:37:45.489153 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.489079    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfwjb\" (UniqueName: \"kubernetes.io/projected/ec1a0361-9b33-4e9c-8566-ecde09242d9e-kube-api-access-cfwjb\") pod \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\" (UID: \"ec1a0361-9b33-4e9c-8566-ecde09242d9e\") "
Jul 07 01:37:45.489418 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.489391    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1a0361-9b33-4e9c-8566-ecde09242d9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec1a0361-9b33-4e9c-8566-ecde09242d9e" (UID: "ec1a0361-9b33-4e9c-8566-ecde09242d9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:37:45.491503 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.491474    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1a0361-9b33-4e9c-8566-ecde09242d9e-kube-api-access-cfwjb" (OuterVolumeSpecName: "kube-api-access-cfwjb") pod "ec1a0361-9b33-4e9c-8566-ecde09242d9e" (UID: "ec1a0361-9b33-4e9c-8566-ecde09242d9e"). InnerVolumeSpecName "kube-api-access-cfwjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:37:45.491641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.491553    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec1a0361-9b33-4e9c-8566-ecde09242d9e" (UID: "ec1a0361-9b33-4e9c-8566-ecde09242d9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:37:45.492139 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.492105    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec1a0361-9b33-4e9c-8566-ecde09242d9e" containerID="e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2" exitCode=0
Jul 07 01:37:45.492248 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.492152    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"
Jul 07 01:37:45.492248 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.492186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8" event={"ID":"ec1a0361-9b33-4e9c-8566-ecde09242d9e","Type":"ContainerDied","Data":"e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2"}
Jul 07 01:37:45.492248 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.492244    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8" event={"ID":"ec1a0361-9b33-4e9c-8566-ecde09242d9e","Type":"ContainerDied","Data":"81e46a3d86d05f45671d7db5482dffc8b9aa7ca519c3781586e2ad3b114c9c55"}
Jul 07 01:37:45.492399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.492273    2562 scope.go:117] "RemoveContainer" containerID="e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2"
Jul 07 01:37:45.512747 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.512724    2562 scope.go:117] "RemoveContainer" containerID="e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2"
Jul 07 01:37:45.513075 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:45.513052    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2\": container with ID starting with e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2 not found: ID does not exist" containerID="e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2"
Jul 07 01:37:45.513135 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.513085    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2"} err="failed to get container status \"e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2\": rpc error: code = NotFound desc = could not find container \"e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2\": container with ID starting with e7ce341e2aaadb03ca60479d20c52460f92042a57dac55b17dbc611885d506a2 not found: ID does not exist"
Jul 07 01:37:45.524636 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.524604    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"]
Jul 07 01:37:45.527310 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.527280    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-df671c-85858bd6d-84wq8"]
Jul 07 01:37:45.590672 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.590632    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a0361-9b33-4e9c-8566-ecde09242d9e-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:45.590672 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.590665    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwjb\" (UniqueName: \"kubernetes.io/projected/ec1a0361-9b33-4e9c-8566-ecde09242d9e-kube-api-access-cfwjb\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:45.590916 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:45.590723    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec1a0361-9b33-4e9c-8566-ecde09242d9e-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:46.289402 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:46.289362    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1a0361-9b33-4e9c-8566-ecde09242d9e" path="/var/lib/kubelet/pods/ec1a0361-9b33-4e9c-8566-ecde09242d9e/volumes"
Jul 07 01:37:46.533419 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:46.533385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:37:49.202414 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.202344    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:49.325976 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.325943    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-data\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.325976 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.325990    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-socket\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-envoy\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326058    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxtlg\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-kube-api-access-hxtlg\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326302 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326252    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-token\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326348 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-credential-socket\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326348 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326253    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:49.326457 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-podinfo\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326457 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-certs\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326457 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326411    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istiod-ca-cert\") pod \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\" (UID: \"e39d1bfe-fe2d-4c16-9595-8020ac2687b0\") "
Jul 07 01:37:49.326780 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326745    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:49.326913 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.326780    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.327193 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.327167    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:49.327259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.327190    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:49.327301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.327263    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:37:49.328525 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.328499    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-kube-api-access-hxtlg" (OuterVolumeSpecName: "kube-api-access-hxtlg") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "kube-api-access-hxtlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:37:49.328609 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.328505    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:37:49.328972 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.328953    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:37:49.328972 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.328953    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e39d1bfe-fe2d-4c16-9595-8020ac2687b0" (UID: "e39d1bfe-fe2d-4c16-9595-8020ac2687b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:37:49.427295 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427244    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.427295 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427288    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.427295 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427298    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.427295 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427307    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.427295 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427314    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.427602 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427322    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxtlg\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-kube-api-access-hxtlg\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.427602 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427330    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.427602 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.427338    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d1bfe-fe2d-4c16-9595-8020ac2687b0-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:37:49.515126 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.515093    2562 generic.go:356] "Generic (PLEG): container finished" podID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerID="a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1" exitCode=0
Jul 07 01:37:49.515345 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.515173    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"
Jul 07 01:37:49.515345 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.515187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" event={"ID":"e39d1bfe-fe2d-4c16-9595-8020ac2687b0","Type":"ContainerDied","Data":"a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1"}
Jul 07 01:37:49.515345 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.515242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp" event={"ID":"e39d1bfe-fe2d-4c16-9595-8020ac2687b0","Type":"ContainerDied","Data":"b59c058032a64f2c3f3325625d6d6280d0f422f2c08b340170c657bc27b44609"}
Jul 07 01:37:49.515345 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.515272    2562 scope.go:117] "RemoveContainer" containerID="a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1"
Jul 07 01:37:49.524445 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.524427    2562 scope.go:117] "RemoveContainer" containerID="d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb"
Jul 07 01:37:49.532817 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.532795    2562 scope.go:117] "RemoveContainer" containerID="a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1"
Jul 07 01:37:49.533111 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:49.533094    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1\": container with ID starting with a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1 not found: ID does not exist" containerID="a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1"
Jul 07 01:37:49.533163 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.533122    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1"} err="failed to get container status \"a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1\": rpc error: code = NotFound desc = could not find container \"a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1\": container with ID starting with a80df5818fca9b1dffaf78b21f3c9a67b88ddb680413fa9428f1ed05b0d6a6c1 not found: ID does not exist"
Jul 07 01:37:49.533163 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.533141    2562 scope.go:117] "RemoveContainer" containerID="d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb"
Jul 07 01:37:49.533411 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:37:49.533397    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb\": container with ID starting with d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb not found: ID does not exist" containerID="d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb"
Jul 07 01:37:49.533456 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.533415    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb"} err="failed to get container status \"d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb\": rpc error: code = NotFound desc = could not find container \"d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb\": container with ID starting with d57a681929571d926713f2deecda635c43575d659274df37a4d89237df8ec0bb not found: ID does not exist"
Jul 07 01:37:49.536263 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.536240    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"]
Jul 07 01:37:49.540214 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:49.540192    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9cd09e-openshift-default-7595786c99-c9dlp"]
Jul 07 01:37:50.290161 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:50.290126    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" path="/var/lib/kubelet/pods/e39d1bfe-fe2d-4c16-9595-8020ac2687b0/volumes"
Jul 07 01:37:50.508335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:50.508295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:50.579565 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:50.579446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:50.759132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:50.759095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:50.999756 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:50.999726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:51.315231 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.315120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:51.577302 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.577207    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"]
Jul 07 01:37:51.578011 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.577984    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy"
Jul 07 01:37:51.578011 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578010    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578024    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5224ffc-5d41-4434-9bb0-75c486c41162" containerName="maas-api"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578033    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5224ffc-5d41-4434-9bb0-75c486c41162" containerName="maas-api"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578044    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578052    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578068    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1a0361-9b33-4e9c-8566-ecde09242d9e" containerName="maas-api"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578076    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1a0361-9b33-4e9c-8566-ecde09242d9e" containerName="maas-api"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578183    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578195    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5224ffc-5d41-4434-9bb0-75c486c41162" containerName="maas-api"
Jul 07 01:37:51.578212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.578208    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1a0361-9b33-4e9c-8566-ecde09242d9e" containerName="maas-api"
Jul 07 01:37:51.582766 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.582746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.585329 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.585311    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 01:37:51.585396 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.585360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ced9d6-openshift-default-dockercfg-rv54w\""
Jul 07 01:37:51.591394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.591364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"]
Jul 07 01:37:51.610022 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.609975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:51.647287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd9350-cdab-4b52-a424-f271730af6f8-istiod-ca-cert\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-credential-socket\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647515 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-istio-token\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647515 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-data\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647515 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-certs\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647515 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-socket\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647515 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8x6d\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-kube-api-access-w8x6d\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647515 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-envoy\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.647731 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.647541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd9350-cdab-4b52-a424-f271730af6f8-istio-podinfo\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.710947 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.710911    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:51.748082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-envoy\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd9350-cdab-4b52-a424-f271730af6f8-istio-podinfo\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd9350-cdab-4b52-a424-f271730af6f8-istiod-ca-cert\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-credential-socket\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-istio-token\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-data\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-certs\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-socket\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748654 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8x6d\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-kube-api-access-w8x6d\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748654 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748576    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-credential-socket\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748654 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748635    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-certs\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748844 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748766    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-socket\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.748844 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.748802    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-data\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.749031 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.749011    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd9350-cdab-4b52-a424-f271730af6f8-istiod-ca-cert\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.750512 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.750494    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-envoy\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.751074 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.751057    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd9350-cdab-4b52-a424-f271730af6f8-istio-podinfo\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.757572 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.757547    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-istio-token\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.758035 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.757995    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8x6d\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-kube-api-access-w8x6d\") pod \"e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") " pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:51.894459 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:51.894362    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:52.038541 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.038512    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"]
Jul 07 01:37:52.040332 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:37:52.040300    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cd9350_cdab_4b52_a424_f271730af6f8.slice/crio-0e78a6fa05023d968c8ddd7dcd12abf6a3877511310867b082984072d12a4048 WatchSource:0}: Error finding container 0e78a6fa05023d968c8ddd7dcd12abf6a3877511310867b082984072d12a4048: Status 404 returned error can't find the container with id 0e78a6fa05023d968c8ddd7dcd12abf6a3877511310867b082984072d12a4048
Jul 07 01:37:52.042626 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.042591    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:52.042735 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.042661    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:52.042735 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.042720    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:37:52.532067 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.532027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" event={"ID":"79cd9350-cdab-4b52-a424-f271730af6f8","Type":"ContainerStarted","Data":"df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307"}
Jul 07 01:37:52.532490 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.532096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" event={"ID":"79cd9350-cdab-4b52-a424-f271730af6f8","Type":"ContainerStarted","Data":"0e78a6fa05023d968c8ddd7dcd12abf6a3877511310867b082984072d12a4048"}
Jul 07 01:37:52.552125 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.552077    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podStartSLOduration=1.55206175 podStartE2EDuration="1.55206175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:37:52.550404917 +0000 UTC m=+2200.825396251" watchObservedRunningTime="2026-07-07 01:37:52.55206175 +0000 UTC m=+2200.827053050"
Jul 07 01:37:52.895044 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:52.894949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:53.900629 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:53.900594    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:54.417740 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.417673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:54.514006 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.513962    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"]
Jul 07 01:37:54.514371 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.514359    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39d1bfe-fe2d-4c16-9595-8020ac2687b0" containerName="istio-proxy"
Jul 07 01:37:54.517410 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.517395    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.520428 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.520395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:54.521047 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.521028    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-ced9d6-serving-cert\""
Jul 07 01:37:54.523460 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.523440    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jtxhz\""
Jul 07 01:37:54.542046 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.542020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:54.543212 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.543189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:37:54.549183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.549156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"]
Jul 07 01:37:54.663043 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.663011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:54.676381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.676294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3e225-56d8-4210-94b7-94d4897bfd4c-maas-api-tls\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.676381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.676352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvvx\" (UniqueName: \"kubernetes.io/projected/daa3e225-56d8-4210-94b7-94d4897bfd4c-kube-api-access-cjvvx\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.676613 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.676488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3e225-56d8-4210-94b7-94d4897bfd4c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.772209 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.772179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:54.777356 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.777329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvvx\" (UniqueName: \"kubernetes.io/projected/daa3e225-56d8-4210-94b7-94d4897bfd4c-kube-api-access-cjvvx\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.777542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.777397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3e225-56d8-4210-94b7-94d4897bfd4c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.777542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.777473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3e225-56d8-4210-94b7-94d4897bfd4c-maas-api-tls\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.778075 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.778054    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3e225-56d8-4210-94b7-94d4897bfd4c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.779952 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.779928    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3e225-56d8-4210-94b7-94d4897bfd4c-maas-api-tls\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.785294 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.785268    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvvx\" (UniqueName: \"kubernetes.io/projected/daa3e225-56d8-4210-94b7-94d4897bfd4c-kube-api-access-cjvvx\") pod \"maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") " pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.830241 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.830205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:54.977002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:54.976864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"]
Jul 07 01:37:54.979547 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:37:54.979515    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa3e225_56d8_4210_94b7_94d4897bfd4c.slice/crio-6d3ba080022766ae520518280074ce6d1b19d6c4f1805195c98fe19ab31e89c3 WatchSource:0}: Error finding container 6d3ba080022766ae520518280074ce6d1b19d6c4f1805195c98fe19ab31e89c3: Status 404 returned error can't find the container with id 6d3ba080022766ae520518280074ce6d1b19d6c4f1805195c98fe19ab31e89c3
Jul 07 01:37:55.066807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:55.066774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:55.167929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:55.167897    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:55.548509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:55.548413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw" event={"ID":"daa3e225-56d8-4210-94b7-94d4897bfd4c","Type":"ContainerStarted","Data":"c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd"}
Jul 07 01:37:55.548509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:55.548477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw" event={"ID":"daa3e225-56d8-4210-94b7-94d4897bfd4c","Type":"ContainerStarted","Data":"6d3ba080022766ae520518280074ce6d1b19d6c4f1805195c98fe19ab31e89c3"}
Jul 07 01:37:55.548759 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:55.548604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:37:55.565137 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:55.565078    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw" podStartSLOduration=1.252034284 podStartE2EDuration="1.56506139s" totalImagesPullingTime="313.027106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:37:55.562987303 +0000 UTC m=+2203.837978626" watchObservedRunningTime="2026-07-07 01:37:55.56506139 +0000 UTC m=+2203.840052711"
Jul 07 01:37:58.918641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:58.918604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:59.235082 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:59.235044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:59.523193 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:59.523105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:37:59.742668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:59.742631    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:37:59.952851 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:37:59.952762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:01.558031 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:01.558003    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:38:02.352954 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.352921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:02.502264 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.502228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:02.572699 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.572643    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"]
Jul 07 01:38:02.576599 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.576575    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.579007 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.578983    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-126d68-serving-cert\""
Jul 07 01:38:02.587716 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.587665    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"]
Jul 07 01:38:02.611167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.611067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:02.749828 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.749781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e72bde7-a510-45ef-9513-402a97b9021c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.750037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.749917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.750037 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.749966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dfw\" (UniqueName: \"kubernetes.io/projected/7e72bde7-a510-45ef-9513-402a97b9021c-kube-api-access-b6dfw\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.850544 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.850497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dfw\" (UniqueName: \"kubernetes.io/projected/7e72bde7-a510-45ef-9513-402a97b9021c-kube-api-access-b6dfw\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.850544 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.850552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e72bde7-a510-45ef-9513-402a97b9021c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.850864 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.850614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.850864 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:38:02.850753    2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-126d68-serving-cert: secret "maas-api-e2e-shared-b-126d68-serving-cert" not found
Jul 07 01:38:02.850864 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:38:02.850824    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls podName:7e72bde7-a510-45ef-9513-402a97b9021c nodeName:}" failed. No retries permitted until 2026-07-07 01:38:03.350803165 +0000 UTC m=+2211.625794474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls") pod "maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" (UID: "7e72bde7-a510-45ef-9513-402a97b9021c") : secret "maas-api-e2e-shared-b-126d68-serving-cert" not found
Jul 07 01:38:02.851345 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.851321    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e72bde7-a510-45ef-9513-402a97b9021c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:02.861584 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:02.861508    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dfw\" (UniqueName: \"kubernetes.io/projected/7e72bde7-a510-45ef-9513-402a97b9021c-kube-api-access-b6dfw\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:03.065718 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:03.065650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:03.166936 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:03.166851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:03.356882 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:03.356838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:03.359474 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:03.359454    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls\") pod \"maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") " pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:03.490928 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:03.490883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:03.680894 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:03.680768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"]
Jul 07 01:38:03.683348 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:38:03.683321    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e72bde7_a510_45ef_9513_402a97b9021c.slice/crio-f2c7d6f55f33ecc8701e848d094c950047d7df40dc65f9072be209e9679a0e0a WatchSource:0}: Error finding container f2c7d6f55f33ecc8701e848d094c950047d7df40dc65f9072be209e9679a0e0a: Status 404 returned error can't find the container with id f2c7d6f55f33ecc8701e848d094c950047d7df40dc65f9072be209e9679a0e0a
Jul 07 01:38:04.594240 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:04.594202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" event={"ID":"7e72bde7-a510-45ef-9513-402a97b9021c","Type":"ContainerStarted","Data":"b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7"}
Jul 07 01:38:04.594458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:04.594258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" event={"ID":"7e72bde7-a510-45ef-9513-402a97b9021c","Type":"ContainerStarted","Data":"f2c7d6f55f33ecc8701e848d094c950047d7df40dc65f9072be209e9679a0e0a"}
Jul 07 01:38:04.594458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:04.594335    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:04.611788 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:04.611733    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" podStartSLOduration=2.236123892 podStartE2EDuration="2.611711629s" totalImagesPullingTime="375.587737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:38:04.610093678 +0000 UTC m=+2212.885085001" watchObservedRunningTime="2026-07-07 01:38:04.611711629 +0000 UTC m=+2212.886702952"
Jul 07 01:38:05.598350 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:05.598316    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:05.703573 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:05.703532    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:06.817597 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:06.817563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:08.617276 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:08.617239    2562 generic.go:356] "Generic (PLEG): container finished" podID="79cd9350-cdab-4b52-a424-f271730af6f8" containerID="df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307" exitCode=0
Jul 07 01:38:08.617884 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:08.617311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" event={"ID":"79cd9350-cdab-4b52-a424-f271730af6f8","Type":"ContainerDied","Data":"df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307"}
Jul 07 01:38:08.617884 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:08.617678    2562 scope.go:117] "RemoveContainer" containerID="df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307"
Jul 07 01:38:08.618865 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:08.618836    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:38:08.618970 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:08.618907    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:38:08.618970 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:08.618939    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 01:38:09.542503 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:09.542458    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:38:09.625537 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:09.625493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" event={"ID":"79cd9350-cdab-4b52-a424-f271730af6f8","Type":"ContainerStarted","Data":"32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057"}
Jul 07 01:38:09.894857 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:09.894749    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:38:09.895861 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:09.895835    2562 patch_prober.go:28] interesting pod/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 07 01:38:09.895936 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:09.895886    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 07 01:38:10.605466 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:10.605438    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:38:10.895048 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:10.894955    2562 patch_prober.go:28] interesting pod/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 07 01:38:10.895048 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:10.895027    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 07 01:38:11.895235 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:11.895193    2562 patch_prober.go:28] interesting pod/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 07 01:38:11.895753 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:11.895268    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 07 01:38:11.955547 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:11.955509    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:38:12.895367 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:12.895323    2562 patch_prober.go:28] interesting pod/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 07 01:38:12.895813 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:12.895382    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 07 01:38:13.891411 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.891376    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"]
Jul 07 01:38:13.895618 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.895589    2562 patch_prober.go:28] interesting pod/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 07 01:38:13.896005 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.895651    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 07 01:38:13.897192 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.897162    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:13.899354 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.899326    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"kube-root-ca.crt\""
Jul 07 01:38:13.900574 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.900553    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"auth-test-model-ced9d6-kserve-self-signed-certs\""
Jul 07 01:38:13.900726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.900581    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"default-dockercfg-54c8g\""
Jul 07 01:38:13.900807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.900742    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"openshift-service-ca.crt\""
Jul 07 01:38:13.903609 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.903585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"]
Jul 07 01:38:13.944216 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.944182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:13.968466 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.968432    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-dshm\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:13.968647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.968474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-tmp-dir\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:13.968647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.968506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-model-cache\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:13.968647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.968622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-home\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:13.968806 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.968651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5p7\" (UniqueName: \"kubernetes.io/projected/3a84470c-59a2-4e43-a076-ba38b26eb180-kube-api-access-kj5p7\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:13.968806 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:13.968707    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a84470c-59a2-4e43-a076-ba38b26eb180-tls-certs\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.069673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.069633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-dshm\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.069673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.069678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-tmp-dir\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.069974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.069743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-model-cache\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.069974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.069815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-home\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.069974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.069839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5p7\" (UniqueName: \"kubernetes.io/projected/3a84470c-59a2-4e43-a076-ba38b26eb180-kube-api-access-kj5p7\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.069974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.069881    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a84470c-59a2-4e43-a076-ba38b26eb180-tls-certs\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.070196 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.070178    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-tmp-dir\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.070275 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.070248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-home\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.070380 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.070308    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-model-cache\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.072189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.072165    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-dshm\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.072649 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.072628    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a84470c-59a2-4e43-a076-ba38b26eb180-tls-certs\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.077812 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.077786    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5p7\" (UniqueName: \"kubernetes.io/projected/3a84470c-59a2-4e43-a076-ba38b26eb180-kube-api-access-kj5p7\") pod \"auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") " pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.208872 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.208774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:14.361059 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.361028    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"]
Jul 07 01:38:14.363019 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:38:14.362989    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a84470c_59a2_4e43_a076_ba38b26eb180.slice/crio-118f296a58857b97563fcd6dfe5a24678dbfdc74a1f464498aca000116457142 WatchSource:0}: Error finding container 118f296a58857b97563fcd6dfe5a24678dbfdc74a1f464498aca000116457142: Status 404 returned error can't find the container with id 118f296a58857b97563fcd6dfe5a24678dbfdc74a1f464498aca000116457142
Jul 07 01:38:14.659753 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.659713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n" event={"ID":"3a84470c-59a2-4e43-a076-ba38b26eb180","Type":"ContainerStarted","Data":"6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2"}
Jul 07 01:38:14.659753 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.659758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n" event={"ID":"3a84470c-59a2-4e43-a076-ba38b26eb180","Type":"ContainerStarted","Data":"118f296a58857b97563fcd6dfe5a24678dbfdc74a1f464498aca000116457142"}
Jul 07 01:38:14.675382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.675327    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n" podStartSLOduration=1.675310823 podStartE2EDuration="1.675310823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:38:14.674732743 +0000 UTC m=+2222.949724066" watchObservedRunningTime="2026-07-07 01:38:14.675310823 +0000 UTC m=+2222.950302144"
Jul 07 01:38:14.899046 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.899016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:38:14.899472 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.899287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:38:14.899985 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:14.899968    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:38:15.766269 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:15.766231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:15.867199 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:15.867110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:21.033047 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:21.033003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:38:23.914715 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:23.914651    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:24.016560 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:24.016522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:24.209370 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:24.209275    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:24.209370 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:24.209333    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:24.223189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:24.223160    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:24.717612 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:24.717581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:38:30.977805 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:30.977763    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"]
Jul 07 01:38:30.982147 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:30.982124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:30.984741 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:30.984709    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"default-dockercfg-zqzw4\""
Jul 07 01:38:30.985005 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:30.984978    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"kube-root-ca.crt\""
Jul 07 01:38:30.985859 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:30.985838    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"openshift-service-ca.crt\""
Jul 07 01:38:30.986158 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:30.986144    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"auth-test-model-126d68-kserve-self-signed-certs\""
Jul 07 01:38:30.989779 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:30.989755    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"]
Jul 07 01:38:31.007274 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.007235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:38:31.019504 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.019467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-model-cache\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.019731 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.019514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9sw9\" (UniqueName: \"kubernetes.io/projected/9f9855f1-192c-42df-a191-9c6dfd8592a9-kube-api-access-c9sw9\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.019731 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.019606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-home\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.019731 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.019661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-dshm\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.019731 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.019729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-tmp-dir\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.019924 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.019760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9855f1-192c-42df-a191-9c6dfd8592a9-tls-certs\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121152 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-home\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-dshm\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-tmp-dir\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9855f1-192c-42df-a191-9c6dfd8592a9-tls-certs\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-model-cache\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sw9\" (UniqueName: \"kubernetes.io/projected/9f9855f1-192c-42df-a191-9c6dfd8592a9-kube-api-access-c9sw9\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121617 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121589    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-home\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121705 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121617    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-tmp-dir\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.121761 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.121731    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-model-cache\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.123793 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.123768    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-dshm\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.123977 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.123959    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9855f1-192c-42df-a191-9c6dfd8592a9-tls-certs\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.130599 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.130574    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9sw9\" (UniqueName: \"kubernetes.io/projected/9f9855f1-192c-42df-a191-9c6dfd8592a9-kube-api-access-c9sw9\") pod \"auth-test-model-126d68-kserve-7c8679bb4c-dbv22\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") " pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.295523 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.295420    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:31.439854 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.439822    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"]
Jul 07 01:38:31.442667 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:38:31.442635    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f9855f1_192c_42df_a191_9c6dfd8592a9.slice/crio-e51cf31ff2d657297795085d45f705d1449b68982678beb3120d5c6f0da8804f WatchSource:0}: Error finding container e51cf31ff2d657297795085d45f705d1449b68982678beb3120d5c6f0da8804f: Status 404 returned error can't find the container with id e51cf31ff2d657297795085d45f705d1449b68982678beb3120d5c6f0da8804f
Jul 07 01:38:31.740571 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.740533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22" event={"ID":"9f9855f1-192c-42df-a191-9c6dfd8592a9","Type":"ContainerStarted","Data":"92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b"}
Jul 07 01:38:31.740787 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.740583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22" event={"ID":"9f9855f1-192c-42df-a191-9c6dfd8592a9","Type":"ContainerStarted","Data":"e51cf31ff2d657297795085d45f705d1449b68982678beb3120d5c6f0da8804f"}
Jul 07 01:38:31.758739 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:31.758669    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22" podStartSLOduration=1.758652927 podStartE2EDuration="1.758652927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:38:31.756357215 +0000 UTC m=+2240.031348548" watchObservedRunningTime="2026-07-07 01:38:31.758652927 +0000 UTC m=+2240.033644271"
Jul 07 01:38:34.341226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:34.341132    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:38:41.296288 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:41.296247    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:41.296288 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:41.296303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:41.308988 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:41.308960    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:41.799896 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:41.799868    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:38:57.849087 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:38:57.849045    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:39:03.901632 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:39:03.901593    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:39:15.749283 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:39:15.749242    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:39:30.343265 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:39:30.343225    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:39:48.338908 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:39:48.338866    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:40:05.785335 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:05.785250    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:40:22.241214 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:22.241173    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:40:33.657147 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:33.657105    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"]
Jul 07 01:40:33.675500 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:33.675465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:40:34.606319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:34.606276    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"]
Jul 07 01:40:34.657419 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:34.657384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:40:38.743548 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:38.743502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:40:48.678816 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.678748    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n" podUID="3a84470c-59a2-4e43-a076-ba38b26eb180" containerName="main" containerID="cri-o://6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2" gracePeriod=45
Jul 07 01:40:48.931399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.931315    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:40:48.970290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970255    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-tmp-dir\") pod \"3a84470c-59a2-4e43-a076-ba38b26eb180\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") "
Jul 07 01:40:48.970290 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970296    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-home\") pod \"3a84470c-59a2-4e43-a076-ba38b26eb180\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") "
Jul 07 01:40:48.970559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-model-cache\") pod \"3a84470c-59a2-4e43-a076-ba38b26eb180\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") "
Jul 07 01:40:48.970559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970370    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-dshm\") pod \"3a84470c-59a2-4e43-a076-ba38b26eb180\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") "
Jul 07 01:40:48.970559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970405    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj5p7\" (UniqueName: \"kubernetes.io/projected/3a84470c-59a2-4e43-a076-ba38b26eb180-kube-api-access-kj5p7\") pod \"3a84470c-59a2-4e43-a076-ba38b26eb180\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") "
Jul 07 01:40:48.970559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970441    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a84470c-59a2-4e43-a076-ba38b26eb180-tls-certs\") pod \"3a84470c-59a2-4e43-a076-ba38b26eb180\" (UID: \"3a84470c-59a2-4e43-a076-ba38b26eb180\") "
Jul 07 01:40:48.970803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970573    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a84470c-59a2-4e43-a076-ba38b26eb180" (UID: "3a84470c-59a2-4e43-a076-ba38b26eb180"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:48.970803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970665    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a84470c-59a2-4e43-a076-ba38b26eb180" (UID: "3a84470c-59a2-4e43-a076-ba38b26eb180"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:48.970803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970703    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-home" (OuterVolumeSpecName: "home") pod "3a84470c-59a2-4e43-a076-ba38b26eb180" (UID: "3a84470c-59a2-4e43-a076-ba38b26eb180"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:48.970803 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970799    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:48.971018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970816    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:48.971018 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.970830    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:48.973069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.973036    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-dshm" (OuterVolumeSpecName: "dshm") pod "3a84470c-59a2-4e43-a076-ba38b26eb180" (UID: "3a84470c-59a2-4e43-a076-ba38b26eb180"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:48.973069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.973049    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a84470c-59a2-4e43-a076-ba38b26eb180-kube-api-access-kj5p7" (OuterVolumeSpecName: "kube-api-access-kj5p7") pod "3a84470c-59a2-4e43-a076-ba38b26eb180" (UID: "3a84470c-59a2-4e43-a076-ba38b26eb180"). InnerVolumeSpecName "kube-api-access-kj5p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:40:48.973069 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:48.973056    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84470c-59a2-4e43-a076-ba38b26eb180-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a84470c-59a2-4e43-a076-ba38b26eb180" (UID: "3a84470c-59a2-4e43-a076-ba38b26eb180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:40:49.072035 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.071992    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a84470c-59a2-4e43-a076-ba38b26eb180-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:49.072035 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.072025    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5p7\" (UniqueName: \"kubernetes.io/projected/3a84470c-59a2-4e43-a076-ba38b26eb180-kube-api-access-kj5p7\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:49.072035 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.072035    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a84470c-59a2-4e43-a076-ba38b26eb180-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:49.098505 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.098467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:40:49.149184 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.149152    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:40:49.425083 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.425041    2562 generic.go:356] "Generic (PLEG): container finished" podID="3a84470c-59a2-4e43-a076-ba38b26eb180" containerID="6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2" exitCode=0
Jul 07 01:40:49.425083 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.425087    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"
Jul 07 01:40:49.425347 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.425090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n" event={"ID":"3a84470c-59a2-4e43-a076-ba38b26eb180","Type":"ContainerDied","Data":"6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2"}
Jul 07 01:40:49.425347 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.425135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n" event={"ID":"3a84470c-59a2-4e43-a076-ba38b26eb180","Type":"ContainerDied","Data":"118f296a58857b97563fcd6dfe5a24678dbfdc74a1f464498aca000116457142"}
Jul 07 01:40:49.425347 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.425158    2562 scope.go:117] "RemoveContainer" containerID="6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2"
Jul 07 01:40:49.434694 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.434657    2562 scope.go:117] "RemoveContainer" containerID="6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2"
Jul 07 01:40:49.435039 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:40:49.435014    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2\": container with ID starting with 6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2 not found: ID does not exist" containerID="6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2"
Jul 07 01:40:49.435090 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.435050    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2"} err="failed to get container status \"6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2\": rpc error: code = NotFound desc = could not find container \"6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2\": container with ID starting with 6d0388e868e9d7eaa0c179d182e8a0f3930518b2b565add549ee5f3f7ccfa5a2 not found: ID does not exist"
Jul 07 01:40:49.445219 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.445184    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"]
Jul 07 01:40:49.448645 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.448618    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/auth-test-model-ced9d6-kserve-6bbbb4c6c-npd7n"]
Jul 07 01:40:49.626974 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.626937    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22" podUID="9f9855f1-192c-42df-a191-9c6dfd8592a9" containerName="main" containerID="cri-o://92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b" gracePeriod=45
Jul 07 01:40:49.886217 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.886191    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:40:49.981400 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981364    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-model-cache\") pod \"9f9855f1-192c-42df-a191-9c6dfd8592a9\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") "
Jul 07 01:40:49.981572 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9855f1-192c-42df-a191-9c6dfd8592a9-tls-certs\") pod \"9f9855f1-192c-42df-a191-9c6dfd8592a9\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") "
Jul 07 01:40:49.981572 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981475    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-home\") pod \"9f9855f1-192c-42df-a191-9c6dfd8592a9\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") "
Jul 07 01:40:49.981572 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-dshm\") pod \"9f9855f1-192c-42df-a191-9c6dfd8592a9\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") "
Jul 07 01:40:49.981572 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9sw9\" (UniqueName: \"kubernetes.io/projected/9f9855f1-192c-42df-a191-9c6dfd8592a9-kube-api-access-c9sw9\") pod \"9f9855f1-192c-42df-a191-9c6dfd8592a9\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") "
Jul 07 01:40:49.981795 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981600    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-tmp-dir\") pod \"9f9855f1-192c-42df-a191-9c6dfd8592a9\" (UID: \"9f9855f1-192c-42df-a191-9c6dfd8592a9\") "
Jul 07 01:40:49.981795 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981766    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f9855f1-192c-42df-a191-9c6dfd8592a9" (UID: "9f9855f1-192c-42df-a191-9c6dfd8592a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:49.981907 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981809    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-home" (OuterVolumeSpecName: "home") pod "9f9855f1-192c-42df-a191-9c6dfd8592a9" (UID: "9f9855f1-192c-42df-a191-9c6dfd8592a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:49.981964 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981949    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f9855f1-192c-42df-a191-9c6dfd8592a9" (UID: "9f9855f1-192c-42df-a191-9c6dfd8592a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:49.982010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.981983    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:49.982010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.982002    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:49.984145 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.984110    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9855f1-192c-42df-a191-9c6dfd8592a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f9855f1-192c-42df-a191-9c6dfd8592a9" (UID: "9f9855f1-192c-42df-a191-9c6dfd8592a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:40:49.984145 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.984127    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9855f1-192c-42df-a191-9c6dfd8592a9-kube-api-access-c9sw9" (OuterVolumeSpecName: "kube-api-access-c9sw9") pod "9f9855f1-192c-42df-a191-9c6dfd8592a9" (UID: "9f9855f1-192c-42df-a191-9c6dfd8592a9"). InnerVolumeSpecName "kube-api-access-c9sw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:40:49.984351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:49.984169    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-dshm" (OuterVolumeSpecName: "dshm") pod "9f9855f1-192c-42df-a191-9c6dfd8592a9" (UID: "9f9855f1-192c-42df-a191-9c6dfd8592a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:40:50.083473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.083412    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9855f1-192c-42df-a191-9c6dfd8592a9-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:50.083473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.083461    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:50.083473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.083473    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9sw9\" (UniqueName: \"kubernetes.io/projected/9f9855f1-192c-42df-a191-9c6dfd8592a9-kube-api-access-c9sw9\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:50.083473 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.083483    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f9855f1-192c-42df-a191-9c6dfd8592a9-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:40:50.290012 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.289920    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a84470c-59a2-4e43-a076-ba38b26eb180" path="/var/lib/kubelet/pods/3a84470c-59a2-4e43-a076-ba38b26eb180/volumes"
Jul 07 01:40:50.432700 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.432641    2562 generic.go:356] "Generic (PLEG): container finished" podID="9f9855f1-192c-42df-a191-9c6dfd8592a9" containerID="92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b" exitCode=0
Jul 07 01:40:50.432873 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.432712    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"
Jul 07 01:40:50.432873 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.432731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22" event={"ID":"9f9855f1-192c-42df-a191-9c6dfd8592a9","Type":"ContainerDied","Data":"92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b"}
Jul 07 01:40:50.432873 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.432775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22" event={"ID":"9f9855f1-192c-42df-a191-9c6dfd8592a9","Type":"ContainerDied","Data":"e51cf31ff2d657297795085d45f705d1449b68982678beb3120d5c6f0da8804f"}
Jul 07 01:40:50.432873 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.432792    2562 scope.go:117] "RemoveContainer" containerID="92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b"
Jul 07 01:40:50.441518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.441499    2562 scope.go:117] "RemoveContainer" containerID="92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b"
Jul 07 01:40:50.441851 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:40:50.441824    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b\": container with ID starting with 92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b not found: ID does not exist" containerID="92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b"
Jul 07 01:40:50.441960 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.441853    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b"} err="failed to get container status \"92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b\": rpc error: code = NotFound desc = could not find container \"92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b\": container with ID starting with 92fb1c7bd80013dc0497959a2bc42956e80c15b6c25b9a00ac3e81bfd917963b not found: ID does not exist"
Jul 07 01:40:50.451962 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.451932    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"]
Jul 07 01:40:50.455232 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:50.455203    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/auth-test-model-126d68-kserve-7c8679bb4c-dbv22"]
Jul 07 01:40:52.290153 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:40:52.290118    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f9855f1-192c-42df-a191-9c6dfd8592a9" path="/var/lib/kubelet/pods/9f9855f1-192c-42df-a191-9c6dfd8592a9/volumes"
Jul 07 01:41:04.375301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.375259    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"]
Jul 07 01:41:04.375798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.375628    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a84470c-59a2-4e43-a076-ba38b26eb180" containerName="main"
Jul 07 01:41:04.375798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.375638    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a84470c-59a2-4e43-a076-ba38b26eb180" containerName="main"
Jul 07 01:41:04.375798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.375658    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f9855f1-192c-42df-a191-9c6dfd8592a9" containerName="main"
Jul 07 01:41:04.375798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.375664    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f9855f1-192c-42df-a191-9c6dfd8592a9" containerName="main"
Jul 07 01:41:04.375798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.375748    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a84470c-59a2-4e43-a076-ba38b26eb180" containerName="main"
Jul 07 01:41:04.375798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.375758    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f9855f1-192c-42df-a191-9c6dfd8592a9" containerName="main"
Jul 07 01:41:04.380113 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.380092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.382816 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.382789    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"kube-root-ca.crt\""
Jul 07 01:41:04.383830 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.383805    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"openshift-service-ca.crt\""
Jul 07 01:41:04.384010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.383849    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"default-dockercfg-54c8g\""
Jul 07 01:41:04.384010 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.383977    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"sub-test-model-ced9d6-kserve-self-signed-certs\""
Jul 07 01:41:04.387244 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.387198    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"]
Jul 07 01:41:04.401111 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.401057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-home\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.401319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.401198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-dshm\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.401319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.401296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-model-cache\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.401426 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.401334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047b3e7e-c02f-42fc-ad76-c101092ce799-tls-certs\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.401521 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.401488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-tmp-dir\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.401595 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.401577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnj7\" (UniqueName: \"kubernetes.io/projected/047b3e7e-c02f-42fc-ad76-c101092ce799-kube-api-access-zfnj7\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.401631 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.401489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:41:04.503057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-model-cache\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503057 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047b3e7e-c02f-42fc-ad76-c101092ce799-tls-certs\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-tmp-dir\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnj7\" (UniqueName: \"kubernetes.io/projected/047b3e7e-c02f-42fc-ad76-c101092ce799-kube-api-access-zfnj7\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-home\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-dshm\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503517 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503447    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-model-cache\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503566 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503509    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-tmp-dir\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.503603 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.503576    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-home\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.505801 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.505770    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-dshm\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.505975 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.505947    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047b3e7e-c02f-42fc-ad76-c101092ce799-tls-certs\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.511288 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.511258    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnj7\" (UniqueName: \"kubernetes.io/projected/047b3e7e-c02f-42fc-ad76-c101092ce799-kube-api-access-zfnj7\") pod \"sub-test-model-ced9d6-kserve-6578d598b5-k5jhh\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") " pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.693969 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.693865    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:04.837567 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:04.837543    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"]
Jul 07 01:41:04.840368 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:41:04.840336    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod047b3e7e_c02f_42fc_ad76_c101092ce799.slice/crio-f5951f47688f6eda3fdf2843f822518614eb15078a5ba622f2bd1f98a1a5a0f6 WatchSource:0}: Error finding container f5951f47688f6eda3fdf2843f822518614eb15078a5ba622f2bd1f98a1a5a0f6: Status 404 returned error can't find the container with id f5951f47688f6eda3fdf2843f822518614eb15078a5ba622f2bd1f98a1a5a0f6
Jul 07 01:41:05.517574 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:05.517520    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh" event={"ID":"047b3e7e-c02f-42fc-ad76-c101092ce799","Type":"ContainerStarted","Data":"2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a"}
Jul 07 01:41:05.518034 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:05.517576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh" event={"ID":"047b3e7e-c02f-42fc-ad76-c101092ce799","Type":"ContainerStarted","Data":"f5951f47688f6eda3fdf2843f822518614eb15078a5ba622f2bd1f98a1a5a0f6"}
Jul 07 01:41:05.534189 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:05.534136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh" podStartSLOduration=1.534121752 podStartE2EDuration="1.534121752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:41:05.533572464 +0000 UTC m=+2393.808563787" watchObservedRunningTime="2026-07-07 01:41:05.534121752 +0000 UTC m=+2393.809113074"
Jul 07 01:41:07.734657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:07.734619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:41:12.346648 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:12.346616    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:41:12.349907 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:12.349886    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t4h9k_d61cb9e3-fa2c-44da-880f-3ba4a2892b47/ovn-acl-logging/0.log"
Jul 07 01:41:14.694796 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:14.694762    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:14.695285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:14.694896    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:14.709414 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:14.709387    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:15.578071 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:15.578038    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:41:20.030250 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:20.030202    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:41:20.134678 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:20.134644    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:41:26.576081 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.576040    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"]
Jul 07 01:41:26.581433 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.581401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.586068 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.586034    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"kube-root-ca.crt\""
Jul 07 01:41:26.586271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.586134    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"default-dockercfg-zqzw4\""
Jul 07 01:41:26.586271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.586193    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"openshift-service-ca.crt\""
Jul 07 01:41:26.586392 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.586315    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"sub-test-model-126d68-kserve-self-signed-certs\""
Jul 07 01:41:26.587351 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.587324    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"]
Jul 07 01:41:26.671297 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.671263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:41:26.707266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.707224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-model-cache\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.707266 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.707268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-home\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.707532 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.707285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f426fd09-c21f-4d90-903f-a31c213f2d47-tls-certs\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.707532 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.707306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hrvs\" (UniqueName: \"kubernetes.io/projected/f426fd09-c21f-4d90-903f-a31c213f2d47-kube-api-access-9hrvs\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.707532 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.707417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-tmp-dir\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.707532 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.707525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-dshm\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-tmp-dir\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-dshm\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-model-cache\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-home\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f426fd09-c21f-4d90-903f-a31c213f2d47-tls-certs\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808394 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hrvs\" (UniqueName: \"kubernetes.io/projected/f426fd09-c21f-4d90-903f-a31c213f2d47-kube-api-access-9hrvs\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808659 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808635    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-tmp-dir\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808759 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808647    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-home\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.808759 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.808728    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-model-cache\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.810807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.810783    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-dshm\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.811098 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.811082    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f426fd09-c21f-4d90-903f-a31c213f2d47-tls-certs\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.817028 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.816994    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hrvs\" (UniqueName: \"kubernetes.io/projected/f426fd09-c21f-4d90-903f-a31c213f2d47-kube-api-access-9hrvs\") pod \"sub-test-model-126d68-kserve-56c8959b9-2xx2x\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") " pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:26.894304 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:26.894210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:27.032065 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:27.031902    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"]
Jul 07 01:41:27.034454 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:41:27.034425    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf426fd09_c21f_4d90_903f_a31c213f2d47.slice/crio-015f9800bf37a054098e74ef2e59a2351b2bd16380edc86829e345afa4b22927 WatchSource:0}: Error finding container 015f9800bf37a054098e74ef2e59a2351b2bd16380edc86829e345afa4b22927: Status 404 returned error can't find the container with id 015f9800bf37a054098e74ef2e59a2351b2bd16380edc86829e345afa4b22927
Jul 07 01:41:27.624301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:27.624265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x" event={"ID":"f426fd09-c21f-4d90-903f-a31c213f2d47","Type":"ContainerStarted","Data":"21bfb08a3899cf6b941beb7375cbd972d4d91ec92905862fdae69217513f96c7"}
Jul 07 01:41:27.624711 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:27.624318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x" event={"ID":"f426fd09-c21f-4d90-903f-a31c213f2d47","Type":"ContainerStarted","Data":"015f9800bf37a054098e74ef2e59a2351b2bd16380edc86829e345afa4b22927"}
Jul 07 01:41:27.641950 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:27.641892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x" podStartSLOduration=1.641876292 podStartE2EDuration="1.641876292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:41:27.639578797 +0000 UTC m=+2415.914570120" watchObservedRunningTime="2026-07-07 01:41:27.641876292 +0000 UTC m=+2415.916867638"
Jul 07 01:41:30.542861 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:30.542819    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:41:36.894848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:36.894731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:36.894848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:36.894802    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:36.907565 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:36.907538    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:37.681869 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:37.681838    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:41:49.737328 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:41:49.737289    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:42:05.345809 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:05.345768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:42:11.882750 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:11.882714    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"]
Jul 07 01:42:11.900348 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:11.900312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:42:12.752486 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:12.752446    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"]
Jul 07 01:42:12.776734 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:12.776662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:42:22.236171 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:22.236126    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:42:26.904220 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:26.904172    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh" podUID="047b3e7e-c02f-42fc-ad76-c101092ce799" containerName="main" containerID="cri-o://2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a" gracePeriod=45
Jul 07 01:42:27.165838 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.165775    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:42:27.357582 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357547    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-dshm\") pod \"047b3e7e-c02f-42fc-ad76-c101092ce799\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") "
Jul 07 01:42:27.357912 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-tmp-dir\") pod \"047b3e7e-c02f-42fc-ad76-c101092ce799\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") "
Jul 07 01:42:27.357912 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357646    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047b3e7e-c02f-42fc-ad76-c101092ce799-tls-certs\") pod \"047b3e7e-c02f-42fc-ad76-c101092ce799\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") "
Jul 07 01:42:27.357912 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-model-cache\") pod \"047b3e7e-c02f-42fc-ad76-c101092ce799\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") "
Jul 07 01:42:27.357912 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357734    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-home\") pod \"047b3e7e-c02f-42fc-ad76-c101092ce799\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") "
Jul 07 01:42:27.357912 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfnj7\" (UniqueName: \"kubernetes.io/projected/047b3e7e-c02f-42fc-ad76-c101092ce799-kube-api-access-zfnj7\") pod \"047b3e7e-c02f-42fc-ad76-c101092ce799\" (UID: \"047b3e7e-c02f-42fc-ad76-c101092ce799\") "
Jul 07 01:42:27.358204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357932    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-model-cache" (OuterVolumeSpecName: "model-cache") pod "047b3e7e-c02f-42fc-ad76-c101092ce799" (UID: "047b3e7e-c02f-42fc-ad76-c101092ce799"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:27.358204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357946    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-home" (OuterVolumeSpecName: "home") pod "047b3e7e-c02f-42fc-ad76-c101092ce799" (UID: "047b3e7e-c02f-42fc-ad76-c101092ce799"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:27.358204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.357960    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "047b3e7e-c02f-42fc-ad76-c101092ce799" (UID: "047b3e7e-c02f-42fc-ad76-c101092ce799"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:27.358204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.358099    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:27.358204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.358117    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:27.358204 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.358132    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:27.360032 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.360010    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047b3e7e-c02f-42fc-ad76-c101092ce799-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "047b3e7e-c02f-42fc-ad76-c101092ce799" (UID: "047b3e7e-c02f-42fc-ad76-c101092ce799"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:42:27.360110 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.360033    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047b3e7e-c02f-42fc-ad76-c101092ce799-kube-api-access-zfnj7" (OuterVolumeSpecName: "kube-api-access-zfnj7") pod "047b3e7e-c02f-42fc-ad76-c101092ce799" (UID: "047b3e7e-c02f-42fc-ad76-c101092ce799"). InnerVolumeSpecName "kube-api-access-zfnj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:42:27.360110 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.360083    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-dshm" (OuterVolumeSpecName: "dshm") pod "047b3e7e-c02f-42fc-ad76-c101092ce799" (UID: "047b3e7e-c02f-42fc-ad76-c101092ce799"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:27.459564 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.459468    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfnj7\" (UniqueName: \"kubernetes.io/projected/047b3e7e-c02f-42fc-ad76-c101092ce799-kube-api-access-zfnj7\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:27.459564 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.459501    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047b3e7e-c02f-42fc-ad76-c101092ce799-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:27.459564 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.459510    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047b3e7e-c02f-42fc-ad76-c101092ce799-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:27.776321 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.776284    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x" podUID="f426fd09-c21f-4d90-903f-a31c213f2d47" containerName="main" containerID="cri-o://21bfb08a3899cf6b941beb7375cbd972d4d91ec92905862fdae69217513f96c7" gracePeriod=45
Jul 07 01:42:27.914757 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.914713    2562 generic.go:356] "Generic (PLEG): container finished" podID="047b3e7e-c02f-42fc-ad76-c101092ce799" containerID="2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a" exitCode=0
Jul 07 01:42:27.915232 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.914784    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"
Jul 07 01:42:27.915232 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.914795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh" event={"ID":"047b3e7e-c02f-42fc-ad76-c101092ce799","Type":"ContainerDied","Data":"2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a"}
Jul 07 01:42:27.915232 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.914840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh" event={"ID":"047b3e7e-c02f-42fc-ad76-c101092ce799","Type":"ContainerDied","Data":"f5951f47688f6eda3fdf2843f822518614eb15078a5ba622f2bd1f98a1a5a0f6"}
Jul 07 01:42:27.915232 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.914865    2562 scope.go:117] "RemoveContainer" containerID="2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a"
Jul 07 01:42:27.916980 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.916956    2562 generic.go:356] "Generic (PLEG): container finished" podID="f426fd09-c21f-4d90-903f-a31c213f2d47" containerID="21bfb08a3899cf6b941beb7375cbd972d4d91ec92905862fdae69217513f96c7" exitCode=0
Jul 07 01:42:27.917103 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.917043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x" event={"ID":"f426fd09-c21f-4d90-903f-a31c213f2d47","Type":"ContainerDied","Data":"21bfb08a3899cf6b941beb7375cbd972d4d91ec92905862fdae69217513f96c7"}
Jul 07 01:42:27.926588 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.926566    2562 scope.go:117] "RemoveContainer" containerID="2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a"
Jul 07 01:42:27.927171 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:42:27.927131    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a\": container with ID starting with 2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a not found: ID does not exist" containerID="2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a"
Jul 07 01:42:27.927301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.927178    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a"} err="failed to get container status \"2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a\": rpc error: code = NotFound desc = could not find container \"2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a\": container with ID starting with 2d29c7f942680baaa8fc189d794916b9d42400796d4159456497f9d377e4111a not found: ID does not exist"
Jul 07 01:42:27.939469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.939437    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"]
Jul 07 01:42:27.944182 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:27.944155    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/sub-test-model-ced9d6-kserve-6578d598b5-k5jhh"]
Jul 07 01:42:28.037774 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.037706    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:42:28.164462 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164430    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-home\") pod \"f426fd09-c21f-4d90-903f-a31c213f2d47\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") "
Jul 07 01:42:28.164656 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164478    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-model-cache\") pod \"f426fd09-c21f-4d90-903f-a31c213f2d47\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") "
Jul 07 01:42:28.164656 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164509    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hrvs\" (UniqueName: \"kubernetes.io/projected/f426fd09-c21f-4d90-903f-a31c213f2d47-kube-api-access-9hrvs\") pod \"f426fd09-c21f-4d90-903f-a31c213f2d47\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") "
Jul 07 01:42:28.164656 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164546    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-dshm\") pod \"f426fd09-c21f-4d90-903f-a31c213f2d47\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") "
Jul 07 01:42:28.164656 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f426fd09-c21f-4d90-903f-a31c213f2d47-tls-certs\") pod \"f426fd09-c21f-4d90-903f-a31c213f2d47\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") "
Jul 07 01:42:28.164894 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164657    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-tmp-dir\") pod \"f426fd09-c21f-4d90-903f-a31c213f2d47\" (UID: \"f426fd09-c21f-4d90-903f-a31c213f2d47\") "
Jul 07 01:42:28.164894 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164744    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-model-cache" (OuterVolumeSpecName: "model-cache") pod "f426fd09-c21f-4d90-903f-a31c213f2d47" (UID: "f426fd09-c21f-4d90-903f-a31c213f2d47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:28.164894 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164757    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-home" (OuterVolumeSpecName: "home") pod "f426fd09-c21f-4d90-903f-a31c213f2d47" (UID: "f426fd09-c21f-4d90-903f-a31c213f2d47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:28.165055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164964    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:28.165055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164967    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f426fd09-c21f-4d90-903f-a31c213f2d47" (UID: "f426fd09-c21f-4d90-903f-a31c213f2d47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:28.165055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.164982    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:28.166896 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.166865    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f426fd09-c21f-4d90-903f-a31c213f2d47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f426fd09-c21f-4d90-903f-a31c213f2d47" (UID: "f426fd09-c21f-4d90-903f-a31c213f2d47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:42:28.167021 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.166946    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f426fd09-c21f-4d90-903f-a31c213f2d47-kube-api-access-9hrvs" (OuterVolumeSpecName: "kube-api-access-9hrvs") pod "f426fd09-c21f-4d90-903f-a31c213f2d47" (UID: "f426fd09-c21f-4d90-903f-a31c213f2d47"). InnerVolumeSpecName "kube-api-access-9hrvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:42:28.167021 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.166953    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-dshm" (OuterVolumeSpecName: "dshm") pod "f426fd09-c21f-4d90-903f-a31c213f2d47" (UID: "f426fd09-c21f-4d90-903f-a31c213f2d47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:42:28.266382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.266344    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hrvs\" (UniqueName: \"kubernetes.io/projected/f426fd09-c21f-4d90-903f-a31c213f2d47-kube-api-access-9hrvs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:28.266382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.266376    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:28.266382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.266388    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f426fd09-c21f-4d90-903f-a31c213f2d47-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:28.266382 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.266397    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f426fd09-c21f-4d90-903f-a31c213f2d47-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:42:28.290209 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.290124    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="047b3e7e-c02f-42fc-ad76-c101092ce799" path="/var/lib/kubelet/pods/047b3e7e-c02f-42fc-ad76-c101092ce799/volumes"
Jul 07 01:42:28.924483 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.924454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"
Jul 07 01:42:28.924968 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.924451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x" event={"ID":"f426fd09-c21f-4d90-903f-a31c213f2d47","Type":"ContainerDied","Data":"015f9800bf37a054098e74ef2e59a2351b2bd16380edc86829e345afa4b22927"}
Jul 07 01:42:28.924968 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.924586    2562 scope.go:117] "RemoveContainer" containerID="21bfb08a3899cf6b941beb7375cbd972d4d91ec92905862fdae69217513f96c7"
Jul 07 01:42:28.940726 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.940671    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"]
Jul 07 01:42:28.942675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:28.942646    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/sub-test-model-126d68-kserve-56c8959b9-2xx2x"]
Jul 07 01:42:30.289167 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:30.289130    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f426fd09-c21f-4d90-903f-a31c213f2d47" path="/var/lib/kubelet/pods/f426fd09-c21f-4d90-903f-a31c213f2d47/volumes"
Jul 07 01:42:37.380367 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.380325    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"]
Jul 07 01:42:37.380837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.380738    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047b3e7e-c02f-42fc-ad76-c101092ce799" containerName="main"
Jul 07 01:42:37.380837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.380751    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="047b3e7e-c02f-42fc-ad76-c101092ce799" containerName="main"
Jul 07 01:42:37.380837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.380779    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f426fd09-c21f-4d90-903f-a31c213f2d47" containerName="main"
Jul 07 01:42:37.380837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.380785    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f426fd09-c21f-4d90-903f-a31c213f2d47" containerName="main"
Jul 07 01:42:37.381000 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.380846    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="047b3e7e-c02f-42fc-ad76-c101092ce799" containerName="main"
Jul 07 01:42:37.381000 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.380859    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f426fd09-c21f-4d90-903f-a31c213f2d47" containerName="main"
Jul 07 01:42:37.385175 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.385149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.390994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.388232    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"kube-root-ca.crt\""
Jul 07 01:42:37.390994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.389268    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"openshift-service-ca.crt\""
Jul 07 01:42:37.390994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.389836    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"rate-test-model-ced9d6-kserve-self-signed-certs\""
Jul 07 01:42:37.390994 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.390841    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ced9d6\"/\"default-dockercfg-54c8g\""
Jul 07 01:42:37.395361 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.395331    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"]
Jul 07 01:42:37.405089 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.405054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:42:37.445179 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.445135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-tmp-dir\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.445372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.445188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-home\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.445372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.445211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-dshm\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.445372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.445268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-model-cache\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.445372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.445308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f7311a-4139-462d-8112-4cfa50670a00-tls-certs\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.445372 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.445333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lns7\" (UniqueName: \"kubernetes.io/projected/c9f7311a-4139-462d-8112-4cfa50670a00-kube-api-access-2lns7\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.546465 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-home\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.546465 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-dshm\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.546807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-model-cache\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.546807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f7311a-4139-462d-8112-4cfa50670a00-tls-certs\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.546807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lns7\" (UniqueName: \"kubernetes.io/projected/c9f7311a-4139-462d-8112-4cfa50670a00-kube-api-access-2lns7\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.546807 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-tmp-dir\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.547028 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546916    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-home\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.547028 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.546971    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-model-cache\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.547123 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.547038    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-tmp-dir\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.549124 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.549104    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-dshm\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.549223 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.549206    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f7311a-4139-462d-8112-4cfa50670a00-tls-certs\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.554509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.554479    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lns7\" (UniqueName: \"kubernetes.io/projected/c9f7311a-4139-462d-8112-4cfa50670a00-kube-api-access-2lns7\") pod \"rate-test-model-ced9d6-kserve-6b8948844c-5wtnf\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") " pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.698908 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.698796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:37.850606 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.850579    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"]
Jul 07 01:42:37.852934 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:42:37.852895    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9f7311a_4139_462d_8112_4cfa50670a00.slice/crio-a2b5d7c5c7cb72ea4b282ae04a316015f0dd8a6ad1233edb819c2f713a10087a WatchSource:0}: Error finding container a2b5d7c5c7cb72ea4b282ae04a316015f0dd8a6ad1233edb819c2f713a10087a: Status 404 returned error can't find the container with id a2b5d7c5c7cb72ea4b282ae04a316015f0dd8a6ad1233edb819c2f713a10087a
Jul 07 01:42:37.969107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.969075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf" event={"ID":"c9f7311a-4139-462d-8112-4cfa50670a00","Type":"ContainerStarted","Data":"d1293491d7e6288b4f21d5d5ed85ed56916fe9784ccca9b568d34b97d5b79f3a"}
Jul 07 01:42:37.969107 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.969119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf" event={"ID":"c9f7311a-4139-462d-8112-4cfa50670a00","Type":"ContainerStarted","Data":"a2b5d7c5c7cb72ea4b282ae04a316015f0dd8a6ad1233edb819c2f713a10087a"}
Jul 07 01:42:37.988145 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:37.988066    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf" podStartSLOduration=0.988042278 podStartE2EDuration="988.042278ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:42:37.985848657 +0000 UTC m=+2486.260839979" watchObservedRunningTime="2026-07-07 01:42:37.988042278 +0000 UTC m=+2486.263033601"
Jul 07 01:42:41.243729 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:41.243663    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:42:44.275301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:44.275268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:42:47.699571 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:47.699530    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:47.700049 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:47.699592    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:47.713930 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:47.713905    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:42:48.027526 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:42:48.027494    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:43:01.983292 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:01.983256    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"]
Jul 07 01:43:01.986446 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:01.986427    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:01.989458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:01.989419    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"kube-root-ca.crt\""
Jul 07 01:43:01.989612 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:01.989428    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"openshift-service-ca.crt\""
Jul 07 01:43:01.990532 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:01.990502    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"default-dockercfg-zqzw4\""
Jul 07 01:43:01.990658 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:01.990580    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-126d68\"/\"rate-test-model-126d68-kserve-self-signed-certs\""
Jul 07 01:43:01.997489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:01.997461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"]
Jul 07 01:43:02.015200 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.015159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:02.065969 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.065924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tls-certs\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.066190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.066010    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-dshm\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.066190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.066050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tmp-dir\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.066190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.066086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pblk9\" (UniqueName: \"kubernetes.io/projected/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-kube-api-access-pblk9\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.066190 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.066124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-home\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.066355 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.066206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-model-cache\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.167513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.167477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tmp-dir\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.167513 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.167524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pblk9\" (UniqueName: \"kubernetes.io/projected/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-kube-api-access-pblk9\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.167832 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.167553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-home\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.167832 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.167576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-model-cache\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.167832 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.167607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tls-certs\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.167832 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.167648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-dshm\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.168045 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.168015    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tmp-dir\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.168104 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.168052    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-home\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.168154 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.168104    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-model-cache\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.170251 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.170226    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-dshm\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.170536 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.170507    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tls-certs\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.176325 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.176287    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pblk9\" (UniqueName: \"kubernetes.io/projected/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-kube-api-access-pblk9\") pod \"rate-test-model-126d68-kserve-df4cc8847-b26db\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") " pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.297833 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.297717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:02.444647 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:02.444622    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"]
Jul 07 01:43:02.448275 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:43:02.448231    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d01bf66_8e39_47ad_a9b3_cfbc19b807e5.slice/crio-d970979c0d2d150038a31e1dd17d80508c438296024c5e012ec1f8cd6618ca86 WatchSource:0}: Error finding container d970979c0d2d150038a31e1dd17d80508c438296024c5e012ec1f8cd6618ca86: Status 404 returned error can't find the container with id d970979c0d2d150038a31e1dd17d80508c438296024c5e012ec1f8cd6618ca86
Jul 07 01:43:03.094287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:03.094251    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db" event={"ID":"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5","Type":"ContainerStarted","Data":"81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2"}
Jul 07 01:43:03.094287 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:03.094295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db" event={"ID":"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5","Type":"ContainerStarted","Data":"d970979c0d2d150038a31e1dd17d80508c438296024c5e012ec1f8cd6618ca86"}
Jul 07 01:43:03.114677 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:03.114619    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db" podStartSLOduration=2.114602389 podStartE2EDuration="2.114602389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:43:03.111907842 +0000 UTC m=+2511.386899164" watchObservedRunningTime="2026-07-07 01:43:03.114602389 +0000 UTC m=+2511.389593710"
Jul 07 01:43:06.255131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:06.255047    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:43:06.470701 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:06.470659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:06.566256 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:06.566155    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:12.298469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:12.298432    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:12.301102 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:12.298494    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:12.311474 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:12.311450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:13.157131 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:13.157100    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:16.466001 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:16.465964    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:17.557557 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:17.557523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:20.442298 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:20.442260    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:43:24.585872 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:24.585834    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:24.686735 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:24.686697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:34.945033 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:34.944991    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:43:42.139249 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:42.139204    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"]
Jul 07 01:43:42.171512 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:42.171477    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:43.059263 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:43.059221    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"]
Jul 07 01:43:43.095754 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:43.095676    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:44.107416 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:44.107377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:44.498371 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:44.498330    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:44.597559 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:44.597522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:44.706111 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:44.706076    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:44.808869 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:44.808755    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"]
Jul 07 01:43:44.809079 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:44.809035    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw" podUID="daa3e225-56d8-4210-94b7-94d4897bfd4c" containerName="maas-api" containerID="cri-o://c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd" gracePeriod=30
Jul 07 01:43:44.846751 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:44.846671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:45.069224 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.069144    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:43:45.154208 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.154166    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3e225-56d8-4210-94b7-94d4897bfd4c-maas-api-tls\") pod \"daa3e225-56d8-4210-94b7-94d4897bfd4c\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") "
Jul 07 01:43:45.154208 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.154212    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3e225-56d8-4210-94b7-94d4897bfd4c-openshift-service-ca\") pod \"daa3e225-56d8-4210-94b7-94d4897bfd4c\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") "
Jul 07 01:43:45.154675 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.154280    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjvvx\" (UniqueName: \"kubernetes.io/projected/daa3e225-56d8-4210-94b7-94d4897bfd4c-kube-api-access-cjvvx\") pod \"daa3e225-56d8-4210-94b7-94d4897bfd4c\" (UID: \"daa3e225-56d8-4210-94b7-94d4897bfd4c\") "
Jul 07 01:43:45.154736 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.154714    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa3e225-56d8-4210-94b7-94d4897bfd4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "daa3e225-56d8-4210-94b7-94d4897bfd4c" (UID: "daa3e225-56d8-4210-94b7-94d4897bfd4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:43:45.156641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.156611    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa3e225-56d8-4210-94b7-94d4897bfd4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "daa3e225-56d8-4210-94b7-94d4897bfd4c" (UID: "daa3e225-56d8-4210-94b7-94d4897bfd4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:43:45.156761 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.156713    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa3e225-56d8-4210-94b7-94d4897bfd4c-kube-api-access-cjvvx" (OuterVolumeSpecName: "kube-api-access-cjvvx") pod "daa3e225-56d8-4210-94b7-94d4897bfd4c" (UID: "daa3e225-56d8-4210-94b7-94d4897bfd4c"). InnerVolumeSpecName "kube-api-access-cjvvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:43:45.255673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.255632    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa3e225-56d8-4210-94b7-94d4897bfd4c-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:45.255673 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.255675    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa3e225-56d8-4210-94b7-94d4897bfd4c-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:45.255964 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.255725    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvvx\" (UniqueName: \"kubernetes.io/projected/daa3e225-56d8-4210-94b7-94d4897bfd4c-kube-api-access-cjvvx\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:45.298730 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.298673    2562 generic.go:356] "Generic (PLEG): container finished" podID="daa3e225-56d8-4210-94b7-94d4897bfd4c" containerID="c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd" exitCode=0
Jul 07 01:43:45.298916 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.298725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw" event={"ID":"daa3e225-56d8-4210-94b7-94d4897bfd4c","Type":"ContainerDied","Data":"c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd"}
Jul 07 01:43:45.298916 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.298759    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"
Jul 07 01:43:45.298916 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.298798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw" event={"ID":"daa3e225-56d8-4210-94b7-94d4897bfd4c","Type":"ContainerDied","Data":"6d3ba080022766ae520518280074ce6d1b19d6c4f1805195c98fe19ab31e89c3"}
Jul 07 01:43:45.298916 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.298829    2562 scope.go:117] "RemoveContainer" containerID="c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd"
Jul 07 01:43:45.309991 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.309963    2562 scope.go:117] "RemoveContainer" containerID="c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd"
Jul 07 01:43:45.310351 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:43:45.310323    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd\": container with ID starting with c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd not found: ID does not exist" containerID="c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd"
Jul 07 01:43:45.310454 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.310367    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd"} err="failed to get container status \"c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd\": rpc error: code = NotFound desc = could not find container \"c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd\": container with ID starting with c67bc4cb9186f6897f66d1593d65f9d4ef07e0268d951456cf2bb8ca26a713cd not found: ID does not exist"
Jul 07 01:43:45.321662 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.321571    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"]
Jul 07 01:43:45.327608 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:45.327571    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ced9d6-5f7dc67c86-fwxhw"]
Jul 07 01:43:46.290252 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:46.290219    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa3e225-56d8-4210-94b7-94d4897bfd4c" path="/var/lib/kubelet/pods/daa3e225-56d8-4210-94b7-94d4897bfd4c/volumes"
Jul 07 01:43:48.586654 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:48.586613    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:43:48.824845 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:48.824800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:49.686826 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:49.686788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:43:57.160509 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.160437    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf" podUID="c9f7311a-4139-462d-8112-4cfa50670a00" containerName="main" containerID="cri-o://d1293491d7e6288b4f21d5d5ed85ed56916fe9784ccca9b568d34b97d5b79f3a" gracePeriod=45
Jul 07 01:43:57.362331 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.362298    2562 generic.go:356] "Generic (PLEG): container finished" podID="c9f7311a-4139-462d-8112-4cfa50670a00" containerID="d1293491d7e6288b4f21d5d5ed85ed56916fe9784ccca9b568d34b97d5b79f3a" exitCode=0
Jul 07 01:43:57.362760 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.362382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf" event={"ID":"c9f7311a-4139-462d-8112-4cfa50670a00","Type":"ContainerDied","Data":"d1293491d7e6288b4f21d5d5ed85ed56916fe9784ccca9b568d34b97d5b79f3a"}
Jul 07 01:43:57.427354 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.427318    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:43:57.566841 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.566804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-dshm\") pod \"c9f7311a-4139-462d-8112-4cfa50670a00\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") "
Jul 07 01:43:57.567055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.566856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-model-cache\") pod \"c9f7311a-4139-462d-8112-4cfa50670a00\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") "
Jul 07 01:43:57.567055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.566886    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-home\") pod \"c9f7311a-4139-462d-8112-4cfa50670a00\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") "
Jul 07 01:43:57.567055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.566920    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f7311a-4139-462d-8112-4cfa50670a00-tls-certs\") pod \"c9f7311a-4139-462d-8112-4cfa50670a00\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") "
Jul 07 01:43:57.567055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.566950    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-tmp-dir\") pod \"c9f7311a-4139-462d-8112-4cfa50670a00\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") "
Jul 07 01:43:57.567055 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.566978    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lns7\" (UniqueName: \"kubernetes.io/projected/c9f7311a-4139-462d-8112-4cfa50670a00-kube-api-access-2lns7\") pod \"c9f7311a-4139-462d-8112-4cfa50670a00\" (UID: \"c9f7311a-4139-462d-8112-4cfa50670a00\") "
Jul 07 01:43:57.567332 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.567141    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9f7311a-4139-462d-8112-4cfa50670a00" (UID: "c9f7311a-4139-462d-8112-4cfa50670a00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:57.567332 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.567217    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9f7311a-4139-462d-8112-4cfa50670a00" (UID: "c9f7311a-4139-462d-8112-4cfa50670a00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:57.567332 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.567230    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-home" (OuterVolumeSpecName: "home") pod "c9f7311a-4139-462d-8112-4cfa50670a00" (UID: "c9f7311a-4139-462d-8112-4cfa50670a00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:57.567332 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.567313    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:57.567530 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.567352    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:57.567530 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.567366    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:57.569319 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.569294    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-dshm" (OuterVolumeSpecName: "dshm") pod "c9f7311a-4139-462d-8112-4cfa50670a00" (UID: "c9f7311a-4139-462d-8112-4cfa50670a00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:57.569442 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.569368    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f7311a-4139-462d-8112-4cfa50670a00-kube-api-access-2lns7" (OuterVolumeSpecName: "kube-api-access-2lns7") pod "c9f7311a-4139-462d-8112-4cfa50670a00" (UID: "c9f7311a-4139-462d-8112-4cfa50670a00"). InnerVolumeSpecName "kube-api-access-2lns7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:43:57.569442 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.569397    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f7311a-4139-462d-8112-4cfa50670a00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9f7311a-4139-462d-8112-4cfa50670a00" (UID: "c9f7311a-4139-462d-8112-4cfa50670a00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:43:57.668188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.668090    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f7311a-4139-462d-8112-4cfa50670a00-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:57.668188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.668127    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lns7\" (UniqueName: \"kubernetes.io/projected/c9f7311a-4139-462d-8112-4cfa50670a00-kube-api-access-2lns7\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:57.668188 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:57.668138    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9f7311a-4139-462d-8112-4cfa50670a00-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:58.080493 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.080434    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db" podUID="6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" containerName="main" containerID="cri-o://81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2" gracePeriod=45
Jul 07 01:43:58.346032 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.345950    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:58.369049 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.369008    2562 generic.go:356] "Generic (PLEG): container finished" podID="6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" containerID="81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2" exitCode=0
Jul 07 01:43:58.369237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.369058    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"
Jul 07 01:43:58.369237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.369097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db" event={"ID":"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5","Type":"ContainerDied","Data":"81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2"}
Jul 07 01:43:58.369237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.369142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db" event={"ID":"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5","Type":"ContainerDied","Data":"d970979c0d2d150038a31e1dd17d80508c438296024c5e012ec1f8cd6618ca86"}
Jul 07 01:43:58.369237 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.369165    2562 scope.go:117] "RemoveContainer" containerID="81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2"
Jul 07 01:43:58.370566 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.370532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf" event={"ID":"c9f7311a-4139-462d-8112-4cfa50670a00","Type":"ContainerDied","Data":"a2b5d7c5c7cb72ea4b282ae04a316015f0dd8a6ad1233edb819c2f713a10087a"}
Jul 07 01:43:58.370714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.370568    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"
Jul 07 01:43:58.379855 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.379809    2562 scope.go:117] "RemoveContainer" containerID="81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2"
Jul 07 01:43:58.380287 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:43:58.380254    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2\": container with ID starting with 81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2 not found: ID does not exist" containerID="81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2"
Jul 07 01:43:58.380430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.380299    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2"} err="failed to get container status \"81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2\": rpc error: code = NotFound desc = could not find container \"81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2\": container with ID starting with 81520c0fdd1dce3c5da448f43e795ed5da5a2e1f619161bf1ef098e9ac4586d2 not found: ID does not exist"
Jul 07 01:43:58.380430 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.380327    2562 scope.go:117] "RemoveContainer" containerID="d1293491d7e6288b4f21d5d5ed85ed56916fe9784ccca9b568d34b97d5b79f3a"
Jul 07 01:43:58.397463 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.397426    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"]
Jul 07 01:43:58.404458 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.404422    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ced9d6/rate-test-model-ced9d6-kserve-6b8948844c-5wtnf"]
Jul 07 01:43:58.474379 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.474320    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-dshm\") pod \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") "
Jul 07 01:43:58.474657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.474419    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pblk9\" (UniqueName: \"kubernetes.io/projected/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-kube-api-access-pblk9\") pod \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") "
Jul 07 01:43:58.474657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.474445    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-model-cache\") pod \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") "
Jul 07 01:43:58.474657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.474493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tls-certs\") pod \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") "
Jul 07 01:43:58.474657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.474509    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-home\") pod \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") "
Jul 07 01:43:58.474657 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.474561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tmp-dir\") pod \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\" (UID: \"6d01bf66-8e39-47ad-a9b3-cfbc19b807e5\") "
Jul 07 01:43:58.474976 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.474831    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" (UID: "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:58.475065 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.475037    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" (UID: "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:58.475120 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.475062    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-home" (OuterVolumeSpecName: "home") pod "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" (UID: "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:58.476943 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.476910    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" (UID: "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:43:58.477381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.477359    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-dshm" (OuterVolumeSpecName: "dshm") pod "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" (UID: "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:43:58.477542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.477521    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-kube-api-access-pblk9" (OuterVolumeSpecName: "kube-api-access-pblk9") pod "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" (UID: "6d01bf66-8e39-47ad-a9b3-cfbc19b807e5"). InnerVolumeSpecName "kube-api-access-pblk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:43:58.576019 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.575977    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pblk9\" (UniqueName: \"kubernetes.io/projected/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-kube-api-access-pblk9\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:58.576019 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.576010    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-model-cache\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:58.576019 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.576020    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tls-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:58.576019 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.576028    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-home\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:58.576019 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.576036    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-tmp-dir\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:58.576334 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.576043    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5-dshm\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:43:58.692226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.692191    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"]
Jul 07 01:43:58.697402 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:43:58.697366    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-126d68/rate-test-model-126d68-kserve-df4cc8847-b26db"]
Jul 07 01:44:00.289710 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:00.289643    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" path="/var/lib/kubelet/pods/6d01bf66-8e39-47ad-a9b3-cfbc19b807e5/volumes"
Jul 07 01:44:00.290120 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:00.290064    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f7311a-4139-462d-8112-4cfa50670a00" path="/var/lib/kubelet/pods/c9f7311a-4139-462d-8112-4cfa50670a00/volumes"
Jul 07 01:44:28.637941 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:28.637899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:28.920463 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:28.920378    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"]
Jul 07 01:44:28.920714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:28.920647    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy" containerID="cri-o://32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057" gracePeriod=30
Jul 07 01:44:28.945075 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:28.945036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:29.161411 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:29.161375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:29.511229 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:29.511187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:29.902187 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:29.902098    2562 patch_prober.go:28] interesting pod/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 01:44:29.902572 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:29.902185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 01:44:29.923301 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:29.923266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:30.237438 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.237395    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:30.413252 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.413216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:30.551379 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.551291    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"]
Jul 07 01:44:30.551552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.551489    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" podUID="7e72bde7-a510-45ef-9513-402a97b9021c" containerName="maas-api" containerID="cri-o://b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7" gracePeriod=30
Jul 07 01:44:30.600792 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.600751    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" podUID="7e72bde7-a510-45ef-9513-402a97b9021c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.56:8443/health\": dial tcp 10.133.0.56:8443: connect: connection refused"
Jul 07 01:44:30.660263 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.660231    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:30.801892 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.801816    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:44:30.848821 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.848787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls\") pod \"7e72bde7-a510-45ef-9513-402a97b9021c\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") "
Jul 07 01:44:30.849025 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.848841    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e72bde7-a510-45ef-9513-402a97b9021c-openshift-service-ca\") pod \"7e72bde7-a510-45ef-9513-402a97b9021c\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") "
Jul 07 01:44:30.849025 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.848918    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6dfw\" (UniqueName: \"kubernetes.io/projected/7e72bde7-a510-45ef-9513-402a97b9021c-kube-api-access-b6dfw\") pod \"7e72bde7-a510-45ef-9513-402a97b9021c\" (UID: \"7e72bde7-a510-45ef-9513-402a97b9021c\") "
Jul 07 01:44:30.849239 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.849215    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e72bde7-a510-45ef-9513-402a97b9021c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e72bde7-a510-45ef-9513-402a97b9021c" (UID: "7e72bde7-a510-45ef-9513-402a97b9021c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:44:30.851309 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.851271    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e72bde7-a510-45ef-9513-402a97b9021c" (UID: "7e72bde7-a510-45ef-9513-402a97b9021c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 01:44:30.851449 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.851427    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e72bde7-a510-45ef-9513-402a97b9021c-kube-api-access-b6dfw" (OuterVolumeSpecName: "kube-api-access-b6dfw") pod "7e72bde7-a510-45ef-9513-402a97b9021c" (UID: "7e72bde7-a510-45ef-9513-402a97b9021c"). InnerVolumeSpecName "kube-api-access-b6dfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:44:30.949921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.949881    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e72bde7-a510-45ef-9513-402a97b9021c-maas-api-tls\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:30.949921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.949916    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e72bde7-a510-45ef-9513-402a97b9021c-openshift-service-ca\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:30.949921 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:30.949927    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6dfw\" (UniqueName: \"kubernetes.io/projected/7e72bde7-a510-45ef-9513-402a97b9021c-kube-api-access-b6dfw\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:31.544708 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.544649    2562 generic.go:356] "Generic (PLEG): container finished" podID="7e72bde7-a510-45ef-9513-402a97b9021c" containerID="b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7" exitCode=0
Jul 07 01:44:31.544925 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.544722    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"
Jul 07 01:44:31.544925 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.544741    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" event={"ID":"7e72bde7-a510-45ef-9513-402a97b9021c","Type":"ContainerDied","Data":"b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7"}
Jul 07 01:44:31.544925 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.544800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd" event={"ID":"7e72bde7-a510-45ef-9513-402a97b9021c","Type":"ContainerDied","Data":"f2c7d6f55f33ecc8701e848d094c950047d7df40dc65f9072be209e9679a0e0a"}
Jul 07 01:44:31.544925 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.544821    2562 scope.go:117] "RemoveContainer" containerID="b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7"
Jul 07 01:44:31.554436 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.554249    2562 scope.go:117] "RemoveContainer" containerID="b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7"
Jul 07 01:44:31.554580 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:44:31.554559    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7\": container with ID starting with b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7 not found: ID does not exist" containerID="b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7"
Jul 07 01:44:31.554624 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.554591    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7"} err="failed to get container status \"b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7\": rpc error: code = NotFound desc = could not find container \"b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7\": container with ID starting with b2e0054fdf727c5362c36cdc07c2910bc2e75886fea3b07981f59e3c211ff7b7 not found: ID does not exist"
Jul 07 01:44:31.570235 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.570200    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"]
Jul 07 01:44:31.584063 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:31.584033    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-126d68-7489684cfc-fl7xd"]
Jul 07 01:44:32.289618 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:32.289581    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e72bde7-a510-45ef-9513-402a97b9021c" path="/var/lib/kubelet/pods/7e72bde7-a510-45ef-9513-402a97b9021c/volumes"
Jul 07 01:44:33.393489 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:33.393450    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2xg84"]
Jul 07 01:44:34.194788 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.194702    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:44:34.281353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281319    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-certs\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281353 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd9350-cdab-4b52-a424-f271730af6f8-istio-podinfo\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281605 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281391    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-data\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281605 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281412    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd9350-cdab-4b52-a424-f271730af6f8-istiod-ca-cert\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281605 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281430    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-istio-token\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281605 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281597    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-envoy\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281640    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-socket\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281677    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8x6d\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-kube-api-access-w8x6d\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-credential-socket\") pod \"79cd9350-cdab-4b52-a424-f271730af6f8\" (UID: \"79cd9350-cdab-4b52-a424-f271730af6f8\") "
Jul 07 01:44:34.281871 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281765    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:44:34.282061 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281897    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cd9350-cdab-4b52-a424-f271730af6f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 01:44:34.282061 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.281973    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:44:34.282156 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.282111    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:44:34.282223 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.282198    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-certs\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.282271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.282227    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd9350-cdab-4b52-a424-f271730af6f8-istiod-ca-cert\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.282271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.282241    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-workload-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.282271 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.282254    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-credential-socket\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.282623 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.282503    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:44:34.284257 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.284230    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:44:34.284425 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.284398    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79cd9350-cdab-4b52-a424-f271730af6f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 01:44:34.284425 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.284415    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 01:44:34.284552 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.284467    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-kube-api-access-w8x6d" (OuterVolumeSpecName: "kube-api-access-w8x6d") pod "79cd9350-cdab-4b52-a424-f271730af6f8" (UID: "79cd9350-cdab-4b52-a424-f271730af6f8"). InnerVolumeSpecName "kube-api-access-w8x6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 01:44:34.383183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.383143    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd9350-cdab-4b52-a424-f271730af6f8-istio-podinfo\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.383183 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.383173    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-data\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.383398 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.383201    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-istio-token\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.383398 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.383210    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd9350-cdab-4b52-a424-f271730af6f8-istio-envoy\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.383398 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.383218    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8x6d\" (UniqueName: \"kubernetes.io/projected/79cd9350-cdab-4b52-a424-f271730af6f8-kube-api-access-w8x6d\") on node \"ip-10-0-142-16.ec2.internal\" DevicePath \"\""
Jul 07 01:44:34.561357 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.561320    2562 generic.go:356] "Generic (PLEG): container finished" podID="79cd9350-cdab-4b52-a424-f271730af6f8" containerID="32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057" exitCode=0
Jul 07 01:44:34.561848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.561403    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"
Jul 07 01:44:34.561848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.561413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" event={"ID":"79cd9350-cdab-4b52-a424-f271730af6f8","Type":"ContainerDied","Data":"32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057"}
Jul 07 01:44:34.561848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.561484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w" event={"ID":"79cd9350-cdab-4b52-a424-f271730af6f8","Type":"ContainerDied","Data":"0e78a6fa05023d968c8ddd7dcd12abf6a3877511310867b082984072d12a4048"}
Jul 07 01:44:34.561848 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.561514    2562 scope.go:117] "RemoveContainer" containerID="32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057"
Jul 07 01:44:34.570930 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.570909    2562 scope.go:117] "RemoveContainer" containerID="df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307"
Jul 07 01:44:34.580080 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.580061    2562 scope.go:117] "RemoveContainer" containerID="32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057"
Jul 07 01:44:34.580468 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:44:34.580447    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057\": container with ID starting with 32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057 not found: ID does not exist" containerID="32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057"
Jul 07 01:44:34.580518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.580483    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057"} err="failed to get container status \"32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057\": rpc error: code = NotFound desc = could not find container \"32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057\": container with ID starting with 32e60d840d3e6cc432210dfb3a303eeb9fac5cb19b3011da4dba97cdf5ea3057 not found: ID does not exist"
Jul 07 01:44:34.580518 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.580503    2562 scope.go:117] "RemoveContainer" containerID="df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307"
Jul 07 01:44:34.580844 ip-10-0-142-16 kubenswrapper[2562]: E0707 01:44:34.580822    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307\": container with ID starting with df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307 not found: ID does not exist" containerID="df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307"
Jul 07 01:44:34.580957 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.580848    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307"} err="failed to get container status \"df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307\": rpc error: code = NotFound desc = could not find container \"df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307\": container with ID starting with df22d1e966cb3f7795743a8cf5e635879f23ca12364ebdb2553c764f5d40c307 not found: ID does not exist"
Jul 07 01:44:34.588201 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.588165    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"]
Jul 07 01:44:34.590399 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:34.590369    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ced9d6-openshift-default-755b4f67f5-tng9w"]
Jul 07 01:44:36.289612 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:36.289574    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" path="/var/lib/kubelet/pods/79cd9350-cdab-4b52-a424-f271730af6f8/volumes"
Jul 07 01:44:37.212381 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:37.212347    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:37.331503 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:37.331471    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:37.542387 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:37.542358    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:37.831644 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:37.831552    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:42.818002 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:42.817962    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gzghl_047b48cc-4194-4c32-bfa0-75dfb2ec0b9f/manager/0.log"
Jul 07 01:44:42.940543 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:42.940514    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m7l6m_76febb9f-b8a4-4fcc-94f7-1c96dda0e81f/manager/0.log"
Jul 07 01:44:43.288846 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:43.288815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723130-8chtv_c822a053-9353-459a-914b-97cd0a1ddae0/cleanup/0.log"
Jul 07 01:44:43.663920 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:43.663884    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6ntnl_3b4ff3e6-74d0-443a-b28a-3a83b971fcd3/manager/0.log"
Jul 07 01:44:43.805241 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:43.805213    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9ffff4fc-45vqz_4f738f51-3152-4dcf-b713-1e147ea2bcb5/copy-manifests/0.log"
Jul 07 01:44:43.816940 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:43.816910    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9ffff4fc-45vqz_4f738f51-3152-4dcf-b713-1e147ea2bcb5/manager/0.log"
Jul 07 01:44:45.671660 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:45.671628    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xgbx9_9458d5fb-093b-49f9-90f2-8fa551735d97/manager/0.log"
Jul 07 01:44:45.793587 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:45.793551    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-t9vd5_95b615bf-4983-4ded-a9df-4e1b999314b1/manager/1.log"
Jul 07 01:44:45.903874 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:45.903841    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z58vn_797adc9d-d61e-45fa-83b3-5e901b21d407/kuadrant-console-plugin/0.log"
Jul 07 01:44:46.030668 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:46.030634    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wgzmx_41ee820d-3002-407c-a183-6bf968208593/registry-server/0.log"
Jul 07 01:44:46.275584 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:46.275554    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2xg84_d75a8c5f-610b-4fbe-bc02-4221167a0eb9/limitador/0.log"
Jul 07 01:44:46.401312 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:46.401219    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x8mn4_dc9ca5e9-8216-464e-83f5-50cdd6c580d4/manager/0.log"
Jul 07 01:44:47.045911 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:47.045882    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b59ff655-j9mqx_8da35937-89d5-48eb-bcb2-ef3d38e15c02/kube-auth-proxy/0.log"
Jul 07 01:44:47.571187 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:47.571156    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-kcpjr_7766b9bb-c719-4447-b7ee-60e0b0d222f2/payload-processing/0.log"
Jul 07 01:44:48.151424 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:48.151388    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-ktl62_5bde9223-820c-4bbe-93de-ae0924d34177/main/0.log"
Jul 07 01:44:48.524730 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:48.524693    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbmtltn_a6c09bc1-dfe4-436a-9eb5-40766354c96b/main/0.log"
Jul 07 01:44:48.805918 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:48.805830    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-m4n6l_53054bed-a718-4372-a0fa-218fc85ace26/main/0.log"
Jul 07 01:44:53.429929 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.429890    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fhw8n/must-gather-7gpjl"]
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430216    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430227    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430248    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f7311a-4139-462d-8112-4cfa50670a00" containerName="main"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430254    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f7311a-4139-462d-8112-4cfa50670a00" containerName="main"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430263    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430268    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430275    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa3e225-56d8-4210-94b7-94d4897bfd4c" containerName="maas-api"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430280    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa3e225-56d8-4210-94b7-94d4897bfd4c" containerName="maas-api"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430286    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e72bde7-a510-45ef-9513-402a97b9021c" containerName="maas-api"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430291    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e72bde7-a510-45ef-9513-402a97b9021c" containerName="maas-api"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430296    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" containerName="main"
Jul 07 01:44:53.430326 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430304    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" containerName="main"
Jul 07 01:44:53.430819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430351    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy"
Jul 07 01:44:53.430819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430360    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e72bde7-a510-45ef-9513-402a97b9021c" containerName="maas-api"
Jul 07 01:44:53.430819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430366    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa3e225-56d8-4210-94b7-94d4897bfd4c" containerName="maas-api"
Jul 07 01:44:53.430819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430374    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d01bf66-8e39-47ad-a9b3-cfbc19b807e5" containerName="main"
Jul 07 01:44:53.430819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430379    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f7311a-4139-462d-8112-4cfa50670a00" containerName="main"
Jul 07 01:44:53.430819 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.430473    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="79cd9350-cdab-4b52-a424-f271730af6f8" containerName="istio-proxy"
Jul 07 01:44:53.433501 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.433479    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.435504 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.435479    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:53.439469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.439446    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fhw8n\"/\"default-dockercfg-gpjhs\""
Jul 07 01:44:53.439469 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.439467    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhw8n\"/\"openshift-service-ca.crt\""
Jul 07 01:44:53.441053 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.441028    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhw8n\"/\"kube-root-ca.crt\""
Jul 07 01:44:53.453442 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.453403    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhw8n/must-gather-7gpjl"]
Jul 07 01:44:53.543259 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.543228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:53.550492 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.550464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k62p\" (UniqueName: \"kubernetes.io/projected/79a6e86e-156d-4c7b-9bc5-379f205dd018-kube-api-access-9k62p\") pod \"must-gather-7gpjl\" (UID: \"79a6e86e-156d-4c7b-9bc5-379f205dd018\") " pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.550595 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.550541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a6e86e-156d-4c7b-9bc5-379f205dd018-must-gather-output\") pod \"must-gather-7gpjl\" (UID: \"79a6e86e-156d-4c7b-9bc5-379f205dd018\") " pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.651888 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.651846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k62p\" (UniqueName: \"kubernetes.io/projected/79a6e86e-156d-4c7b-9bc5-379f205dd018-kube-api-access-9k62p\") pod \"must-gather-7gpjl\" (UID: \"79a6e86e-156d-4c7b-9bc5-379f205dd018\") " pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.652094 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.651937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a6e86e-156d-4c7b-9bc5-379f205dd018-must-gather-output\") pod \"must-gather-7gpjl\" (UID: \"79a6e86e-156d-4c7b-9bc5-379f205dd018\") " pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.652300 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.652279    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/79a6e86e-156d-4c7b-9bc5-379f205dd018-must-gather-output\") pod \"must-gather-7gpjl\" (UID: \"79a6e86e-156d-4c7b-9bc5-379f205dd018\") " pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.664147 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.664118    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k62p\" (UniqueName: \"kubernetes.io/projected/79a6e86e-156d-4c7b-9bc5-379f205dd018-kube-api-access-9k62p\") pod \"must-gather-7gpjl\" (UID: \"79a6e86e-156d-4c7b-9bc5-379f205dd018\") " pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.752192 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.752146    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhw8n/must-gather-7gpjl"
Jul 07 01:44:53.907053 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.906957    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhw8n/must-gather-7gpjl"]
Jul 07 01:44:53.909866 ip-10-0-142-16 kubenswrapper[2562]: W0707 01:44:53.909838    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79a6e86e_156d_4c7b_9bc5_379f205dd018.slice/crio-de5c3035ebcbdf4d7db17ee00ea72c8c64ebf7618e0492994946957dafc36d78 WatchSource:0}: Error finding container de5c3035ebcbdf4d7db17ee00ea72c8c64ebf7618e0492994946957dafc36d78: Status 404 returned error can't find the container with id de5c3035ebcbdf4d7db17ee00ea72c8c64ebf7618e0492994946957dafc36d78
Jul 07 01:44:53.911752 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:53.911735    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 01:44:54.396996 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:54.396953    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log" limit=1024
Jul 07 01:44:54.657090 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:54.657012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhw8n/must-gather-7gpjl" event={"ID":"79a6e86e-156d-4c7b-9bc5-379f205dd018","Type":"ContainerStarted","Data":"de5c3035ebcbdf4d7db17ee00ea72c8c64ebf7618e0492994946957dafc36d78"}
Jul 07 01:44:55.665894 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:55.665862    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhw8n/must-gather-7gpjl" event={"ID":"79a6e86e-156d-4c7b-9bc5-379f205dd018","Type":"ContainerStarted","Data":"f24c7cb1831f82df9fb9e71201432f01226b4133f0b2ac051dd18b2454f66b58"}
Jul 07 01:44:55.666306 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:55.665904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhw8n/must-gather-7gpjl" event={"ID":"79a6e86e-156d-4c7b-9bc5-379f205dd018","Type":"ContainerStarted","Data":"dd1f5412ea47b0c924eede3ebd609f5ba399d0402166158f8abed7ac97ee4d65"}
Jul 07 01:44:55.686132 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:55.686062    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fhw8n/must-gather-7gpjl" podStartSLOduration=1.8833813529999999 podStartE2EDuration="2.686037903s" totalImagesPullingTime="802.65655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 01:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:44:55.682506412 +0000 UTC m=+2623.957497726" watchObservedRunningTime="2026-07-07 01:44:55.686037903 +0000 UTC m=+2623.961029227"
Jul 07 01:44:56.453837 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:56.453800    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lqst9_85ba5407-90de-4401-a47c-51227f2e1958/global-pull-secret-syncer/0.log"
Jul 07 01:44:56.509532 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:56.509502    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ddjh7_46bdeae8-a38b-4939-adbf-91f694cad4ca/konnectivity-agent/0.log"
Jul 07 01:44:56.678627 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:44:56.678600    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-16.ec2.internal_a927f5a1e989d40b4e926ec92d449add/haproxy/0.log"
Jul 07 01:45:02.025270 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:02.025179    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xgbx9_9458d5fb-093b-49f9-90f2-8fa551735d97/manager/0.log"
Jul 07 01:45:02.070322 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:02.069511    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-t9vd5_95b615bf-4983-4ded-a9df-4e1b999314b1/manager/0.log"
Jul 07 01:45:02.070322 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:02.070223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-t9vd5_95b615bf-4983-4ded-a9df-4e1b999314b1/manager/1.log"
Jul 07 01:45:02.127393 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:02.127358    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z58vn_797adc9d-d61e-45fa-83b3-5e901b21d407/kuadrant-console-plugin/0.log"
Jul 07 01:45:02.182798 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:02.182755    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wgzmx_41ee820d-3002-407c-a183-6bf968208593/registry-server/0.log"
Jul 07 01:45:02.319988 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:02.319900    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2xg84_d75a8c5f-610b-4fbe-bc02-4221167a0eb9/limitador/0.log"
Jul 07 01:45:02.413405 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:02.413343    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x8mn4_dc9ca5e9-8216-464e-83f5-50cdd6c580d4/manager/0.log"
Jul 07 01:45:04.806976 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:04.806944    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpxgb_a3da0820-4c8d-4380-b8fe-0c360f2d45c3/node-exporter/0.log"
Jul 07 01:45:04.830302 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:04.830272    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpxgb_a3da0820-4c8d-4380-b8fe-0c360f2d45c3/kube-rbac-proxy/0.log"
Jul 07 01:45:04.853446 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:04.853414    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mpxgb_a3da0820-4c8d-4380-b8fe-0c360f2d45c3/init-textfile/0.log"
Jul 07 01:45:06.571035 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.570998    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"]
Jul 07 01:45:06.576020 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.575990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.585772 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.585604    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"]
Jul 07 01:45:06.681285 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.681236    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st5sm\" (UniqueName: \"kubernetes.io/projected/3ddc4a6a-e739-4d27-bfde-83d0b5210391-kube-api-access-st5sm\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.681488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.681318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-proc\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.681488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.681347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-podres\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.681488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.681394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-lib-modules\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.681488 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.681420    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-sys\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782325 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st5sm\" (UniqueName: \"kubernetes.io/projected/3ddc4a6a-e739-4d27-bfde-83d0b5210391-kube-api-access-st5sm\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-proc\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-podres\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-lib-modules\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-sys\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782464    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-proc\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782775 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782544    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-sys\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782775 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782587    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-podres\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.782775 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.782587    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ddc4a6a-e739-4d27-bfde-83d0b5210391-lib-modules\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.791865 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.791794    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st5sm\" (UniqueName: \"kubernetes.io/projected/3ddc4a6a-e739-4d27-bfde-83d0b5210391-kube-api-access-st5sm\") pod \"perf-node-gather-daemonset-rprrs\" (UID: \"3ddc4a6a-e739-4d27-bfde-83d0b5210391\") " pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:06.891542 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:06.891450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:07.100716 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:07.100592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"]
Jul 07 01:45:07.749004 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:07.748425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs" event={"ID":"3ddc4a6a-e739-4d27-bfde-83d0b5210391","Type":"ContainerStarted","Data":"18521dc04a61d73c9a7395c33d378ff5ccdb9e30f6f65867c0911797bc48dcf0"}
Jul 07 01:45:07.749004 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:07.748497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs" event={"ID":"3ddc4a6a-e739-4d27-bfde-83d0b5210391","Type":"ContainerStarted","Data":"583e6d6d4125987dc01dcd7de8fef81df8c927e2c1a0944503502053f1992e80"}
Jul 07 01:45:07.749004 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:07.748528    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:07.765456 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:07.765394    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs" podStartSLOduration=1.765369813 podStartE2EDuration="1.765369813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 01:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 01:45:07.763612903 +0000 UTC m=+2636.038604252" watchObservedRunningTime="2026-07-07 01:45:07.765369813 +0000 UTC m=+2636.040361135"
Jul 07 01:45:09.570724 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:09.570677    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnjxx_a0b885b9-6b31-4d8a-948e-bf60f787f84b/dns/0.log"
Jul 07 01:45:09.594641 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:09.594603    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pnjxx_a0b885b9-6b31-4d8a-948e-bf60f787f84b/kube-rbac-proxy/0.log"
Jul 07 01:45:09.669106 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:09.669072    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-75844_aa786802-732e-45a6-9420-7cfe2571fd09/dns-node-resolver/0.log"
Jul 07 01:45:10.289357 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:10.289314    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-97djw_b2d23b4d-300b-4264-b083-6172651d2548/node-ca/0.log"
Jul 07 01:45:11.554888 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:11.554856    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b59ff655-j9mqx_8da35937-89d5-48eb-bcb2-ef3d38e15c02/kube-auth-proxy/0.log"
Jul 07 01:45:11.753714 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:11.753648    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-kcpjr_7766b9bb-c719-4447-b7ee-60e0b0d222f2/payload-processing/0.log"
Jul 07 01:45:12.499203 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:12.499167    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8g4vr_2e1d0fd7-9197-4c07-91ce-852454ca238b/serve-healthcheck-canary/0.log"
Jul 07 01:45:13.032616 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:13.032589    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlgwz_b3d91970-b381-4618-b90b-3de5e92be502/kube-rbac-proxy/0.log"
Jul 07 01:45:13.056171 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:13.056142    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlgwz_b3d91970-b381-4618-b90b-3de5e92be502/exporter/0.log"
Jul 07 01:45:13.082762 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:13.082727    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlgwz_b3d91970-b381-4618-b90b-3de5e92be502/extractor/0.log"
Jul 07 01:45:13.768245 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:13.768211    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fhw8n/perf-node-gather-daemonset-rprrs"
Jul 07 01:45:15.503205 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:15.503142    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gzghl_047b48cc-4194-4c32-bfa0-75dfb2ec0b9f/manager/0.log"
Jul 07 01:45:15.681460 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:15.681427    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m7l6m_76febb9f-b8a4-4fcc-94f7-1c96dda0e81f/manager/0.log"
Jul 07 01:45:15.764759 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:15.764725    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723130-8chtv_c822a053-9353-459a-914b-97cd0a1ddae0/cleanup/0.log"
Jul 07 01:45:16.008200 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:16.008162    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6ntnl_3b4ff3e6-74d0-443a-b28a-3a83b971fcd3/manager/0.log"
Jul 07 01:45:16.040070 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:16.040030    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9ffff4fc-45vqz_4f738f51-3152-4dcf-b713-1e147ea2bcb5/manager/0.log"
Jul 07 01:45:16.062743 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:16.062716    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9ffff4fc-45vqz_4f738f51-3152-4dcf-b713-1e147ea2bcb5/copy-manifests/0.log"
Jul 07 01:45:17.725226 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:17.725200    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6bt2f_e447403b-0948-4743-b33c-d493bc7af52c/openshift-lws-operator/0.log"
Jul 07 01:45:18.024979 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:18.024897    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b59ff655-j9mqx_8da35937-89d5-48eb-bcb2-ef3d38e15c02/kube-auth-proxy/0.log"
Jul 07 01:45:18.192674 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:18.192631    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-kcpjr_7766b9bb-c719-4447-b7ee-60e0b0d222f2/payload-processing/0.log"
Jul 07 01:45:19.608144 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:19.608104    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9b59ff655-j9mqx_8da35937-89d5-48eb-bcb2-ef3d38e15c02/kube-auth-proxy/0.log"
Jul 07 01:45:19.774736 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:19.774704    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-kcpjr_7766b9bb-c719-4447-b7ee-60e0b0d222f2/payload-processing/0.log"
Jul 07 01:45:24.217825 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:24.217801    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fkb2_8216316d-d40e-4cf4-8d4c-13974c5e5f74/kube-multus-additional-cni-plugins/0.log"
Jul 07 01:45:24.241414 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:24.241386    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fkb2_8216316d-d40e-4cf4-8d4c-13974c5e5f74/egress-router-binary-copy/0.log"
Jul 07 01:45:24.263510 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:24.263483    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fkb2_8216316d-d40e-4cf4-8d4c-13974c5e5f74/cni-plugins/0.log"
Jul 07 01:45:24.284843 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:24.284810    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fkb2_8216316d-d40e-4cf4-8d4c-13974c5e5f74/bond-cni-plugin/0.log"
Jul 07 01:45:24.311456 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:24.311426    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fkb2_8216316d-d40e-4cf4-8d4c-13974c5e5f74/routeoverride-cni/0.log"
Jul 07 01:45:24.334556 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:24.334513    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fkb2_8216316d-d40e-4cf4-8d4c-13974c5e5f74/whereabouts-cni-bincopy/0.log"
Jul 07 01:45:24.371419 ip-10-0-142-16 kubenswrapper[2562]: I0707 01:45:24.371390    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fkb2_8216316d-d40e-4cf4-8d4c-13974c5e5f74/whereabouts-cni/0.log"