Jul 27 21:33:50.061150 ip-10-0-133-37 systemd[1]: Starting Kubernetes Kubelet... Jul 27 21:33:50.452384 ip-10-0-133-37 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:33:50.452384 ip-10-0-133-37 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 21:33:50.452384 ip-10-0-133-37 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:33:50.452384 ip-10-0-133-37 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 21:33:50.452384 ip-10-0-133-37 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 21:33:50.454148 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.453997 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 21:33:50.457496 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457478 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:33:50.457496 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457496 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457500 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457504 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457507 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457510 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457513 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457516 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457519 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457521 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457524 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457527 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457530 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457533 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457535 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457538 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457541 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457543 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457545 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457548 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457551 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:33:50.457571 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457553 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457556 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457558 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457561 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457563 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457566 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457569 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457572 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457574 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457584 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457588 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457590 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457593 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457595 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457598 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457602 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457605 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457608 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457610 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457613 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:33:50.458069 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457615 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457618 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457620 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457623 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457626 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457628 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457631 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457633 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457636 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457639 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457641 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457644 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457646 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457649 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457652 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457655 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457658 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457660 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457663 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457666 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:33:50.458588 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457668 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457671 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457674 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457677 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457680 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457683 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457688 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457691 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457694 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457698 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457702 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457705 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457708 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457710 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457713 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457716 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457718 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457721 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:33:50.459089 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457725 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457729 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457734 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457737 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457739 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457742 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457744 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457747 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457750 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457753 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457756 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457759 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457762 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457764 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457767 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457770 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457772 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457775 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457778 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.457781 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:33:50.459524 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458510 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458517 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458520 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458523 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458526 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458529 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458531 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458534 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458537 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458540 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458543 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458545 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458548 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458551 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458555 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458557 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458561 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458563 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458566 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:33:50.460025 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458569 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458572 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458575 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458578 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458580 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458583 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458586 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458588 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458591 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458594 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458597 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458600 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458603 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458606 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458608 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458611 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458614 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458616 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458620 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:33:50.460492 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458622 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458625 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458627 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458630 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458632 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458635 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458637 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458642 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458646 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458650 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458654 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458657 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458659 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458662 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458665 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458668 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458670 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458673 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458676 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458678 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:33:50.460986 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458681 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458684 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458687 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458690 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458692 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458694 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458697 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458701 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458705 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458708 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458710 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458714 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458717 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458720 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458723 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458725 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458729 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458732 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:33:50.461461 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458735 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458738 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458741 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458744 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458747 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458750 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458752 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458755 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458758 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458760 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458763 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458766 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458768 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458771 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458773 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458776 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458778 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458781 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458784 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458786 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458789 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:33:50.461929 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458791 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.458794 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459198 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459204 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459207 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459210 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459212 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459215 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459218 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459220 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459223 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459226 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459229 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459232 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459234 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459237 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459240 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459242 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459245 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459248 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:33:50.462454 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459250 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459253 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459256 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459259 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459261 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459264 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459267 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459269 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459271 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459274 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459284 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459288 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459291 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459293 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459296 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459300 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459303 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459306 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459308 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:33:50.462962 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459311 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459313 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459316 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459318 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459321 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459323 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459326 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459328 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459330 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459333 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459336 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459339 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459344 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459346 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459349 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459351 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459354 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459356 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459361 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:33:50.463463 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459364 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459367 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459370 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459373 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459376 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459379 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459382 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459384 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459395 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459399 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459402 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459404 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459407 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459409 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459412 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459414 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459418 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459422 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459425 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:33:50.463971 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459427 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459430 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459433 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459436 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459439 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459442 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459447 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459450 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459453 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459455 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459458 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459461 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459463 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459466 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459469 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459473 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459476 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459478 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459481 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459483 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:33:50.464433 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459486 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459488 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459491 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.459493 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459572 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459580 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459587 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459592 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459601 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459605 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459609 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459613 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459616 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459619 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459623 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459626 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459629 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459633 2556 flags.go:64] FLAG: --cgroup-root="" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459636 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459639 2556 flags.go:64] FLAG: --client-ca-file="" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459642 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459645 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459649 2556 flags.go:64] FLAG: --cluster-domain="" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459652 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 21:33:50.464941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459655 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459658 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459661 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459667 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459670 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459673 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459676 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459679 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459682 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459685 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459688 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459691 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459695 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459698 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459701 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459704 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459707 2556 flags.go:64] FLAG: --enable-server="true" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459710 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459714 2556 flags.go:64] FLAG: --event-burst="100" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459717 2556 flags.go:64] FLAG: --event-qps="50" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459720 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459723 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459726 2556 flags.go:64] FLAG: --eviction-hard="" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459730 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 21:33:50.465549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459733 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459736 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459740 2556 flags.go:64] FLAG: --eviction-soft="" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459744 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459746 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459749 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459752 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459755 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459758 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459761 2556 flags.go:64] FLAG: --feature-gates="" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459765 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459769 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459772 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459775 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459778 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459781 2556 flags.go:64] FLAG: --help="false" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459784 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459787 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459790 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459793 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459796 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459799 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459802 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 21:33:50.466365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459805 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459808 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459811 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459814 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459817 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459820 2556 flags.go:64] FLAG: --kube-reserved="" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459822 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459825 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459828 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459831 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459834 2556 flags.go:64] FLAG: --lock-file="" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459837 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459841 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459857 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459863 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459866 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459869 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459872 2556 flags.go:64] FLAG: --logging-format="text" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459876 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459879 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459883 2556 flags.go:64] FLAG: --manifest-url="" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459886 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459891 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459894 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459898 2556 flags.go:64] FLAG: --max-pods="110" Jul 27 21:33:50.467399 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459901 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459904 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459907 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459910 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459913 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459916 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459919 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459926 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459930 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459933 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459936 2556 flags.go:64] FLAG: --pod-cidr="" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459939 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459945 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459948 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459951 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459954 2556 flags.go:64] FLAG: --port="10250" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459957 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459960 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e600912956fd001a" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459963 2556 flags.go:64] FLAG: --qos-reserved="" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459967 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459970 2556 flags.go:64] FLAG: --register-node="true" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459973 2556 flags.go:64] FLAG: --register-with-taints="" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459977 2556 flags.go:64] FLAG: --registry-burst="10" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459980 2556 flags.go:64] FLAG: --registry-qps="5" Jul 27 21:33:50.468507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459983 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459986 2556 flags.go:64] FLAG: --reserved-memory="" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459989 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459994 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.459997 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460000 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460003 2556 flags.go:64] FLAG: --runonce="false" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460006 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460009 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460011 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460014 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460017 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460020 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460023 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460026 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460029 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460032 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460035 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460039 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460042 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460045 2556 flags.go:64] FLAG: --system-cgroups="" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460047 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460053 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460055 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460058 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 21:33:50.469562 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460063 2556 flags.go:64] FLAG: --tls-min-version="" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460065 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460072 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460075 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460078 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460081 2556 flags.go:64] FLAG: --v="2" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460085 2556 flags.go:64] FLAG: --version="false" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460089 2556 flags.go:64] FLAG: --vmodule="" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460093 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460096 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460213 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460216 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460219 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460222 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460225 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460228 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460231 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460233 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460236 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460239 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460241 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:33:50.470382 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460243 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460248 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460251 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460254 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460257 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460260 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460263 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460265 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460268 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460270 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460273 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460275 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460278 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460282 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460285 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460289 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460293 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460296 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460299 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:33:50.471229 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460301 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460304 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460307 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460309 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460312 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460315 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460317 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460320 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460322 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460325 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460327 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460330 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460332 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460334 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460337 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460339 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460342 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460344 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460347 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460349 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:33:50.472078 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460352 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460355 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460357 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460359 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460362 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460364 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460368 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460370 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460373 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460375 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460378 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460381 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460383 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460386 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460388 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460391 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460393 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460397 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460399 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460402 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:33:50.472610 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460404 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460407 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460409 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460412 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460414 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460417 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460419 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460422 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460425 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460427 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460429 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460432 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460434 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460437 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460440 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460442 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460445 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460447 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460451 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460454 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:33:50.473337 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460456 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460460 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460463 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460466 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460469 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460471 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460474 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460477 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.460479 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.460485 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.468291 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.468315 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.468345 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.468355 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468446 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:33:50.473873 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468456 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468460 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468465 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468470 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468474 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468479 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468483 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468488 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468492 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468496 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468500 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468504 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468508 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468512 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468516 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468519 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468523 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468527 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468531 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468534 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:33:50.474258 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468539 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468543 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468547 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468552 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468556 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468560 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468565 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468569 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468582 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468587 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468591 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468596 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468600 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468605 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468609 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468613 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468617 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468622 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468626 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:33:50.474804 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468631 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468635 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468639 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468643 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468647 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468652 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468656 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468661 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468665 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468669 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468673 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468678 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468684 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468690 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468695 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468699 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468703 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468708 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468712 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468716 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:33:50.475362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468721 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468726 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468737 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468742 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468746 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468750 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468754 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468758 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468762 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468766 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468771 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468775 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468779 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468783 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468788 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468791 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468796 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468800 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468804 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:33:50.475875 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468808 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468814 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468820 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468824 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468829 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468833 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468837 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468862 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468869 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468874 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468879 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468885 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468890 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468895 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468900 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468904 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468916 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468921 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468926 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:33:50.476376 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.468930 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.468939 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469175 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469184 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469190 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469195 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469199 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469203 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469208 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469212 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469216 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469220 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469224 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469228 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469232 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 21:33:50.477002 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469236 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469241 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469245 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469249 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469254 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469258 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469262 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469266 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469270 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469274 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469278 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469283 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469287 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469291 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469295 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469311 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469316 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469320 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469324 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469328 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 21:33:50.477617 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469332 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469336 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469340 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469344 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469348 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469353 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469356 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469363 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469370 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469374 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469379 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469383 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469388 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469392 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469396 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469401 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469405 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469410 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469414 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 21:33:50.478264 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469418 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469422 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469426 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469431 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469435 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469439 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469443 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469447 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469452 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469467 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469472 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469476 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469480 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469484 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469488 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469494 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469500 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469504 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469508 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469512 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 21:33:50.478885 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469517 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469521 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469525 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469529 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469533 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469538 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469542 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469546 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469550 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469555 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469559 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469563 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469567 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469571 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469575 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469579 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469586 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469592 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469597 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 21:33:50.479605 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469602 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469608 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469612 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469624 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469629 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469633 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469637 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:50.469642 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.469650 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.469840 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.472696 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.474504 2556 server.go:1019] "Starting client certificate rotation" Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.474614 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.474650 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 21:33:50.480230 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.477513 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:33:50.488903 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.488878 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 21:33:50.501630 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.501606 2556 log.go:25] "Validated CRI v1 runtime API" Jul 27 21:33:50.501923 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.501903 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:33:50.503700 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.503683 2556 log.go:25] "Validated CRI v1 image API" Jul 27 21:33:50.504567 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.504552 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 21:33:50.510067 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.510043 2556 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a124fb5-c416-42ca-a4b9-95d654a05a20:/dev/nvme0n1p3 98a82f18-571c-4642-a208-f2a0ff8db25a:/dev/nvme0n1p4] Jul 27 21:33:50.510139 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.510065 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 21:33:50.515612 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.515494 2556 manager.go:216] Machine: {Timestamp:2026-07-27 21:33:50.514195199 +0000 UTC m=+0.337441520 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2974681 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec208a120e28855113a6b38091ba4373 SystemUUID:ec208a12-0e28-8551-13a6-b38091ba4373 BootID:69bdca39-e9a5-4795-9375-21e7d6ff6347 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5f:52:b9:88:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5f:52:b9:88:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:9c:04:e3:a0:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 21:33:50.515612 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.515612 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 21:33:50.515724 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.515707 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 21:33:50.518416 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.518389 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 21:33:50.518572 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.518419 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 21:33:50.519175 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.519163 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 21:33:50.519207 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.519178 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 21:33:50.519207 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.519186 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:33:50.519755 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.519744 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 21:33:50.520866 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.520837 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 21:33:50.524491 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.524477 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:33:50.524670 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.524660 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 21:33:50.524706 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.524696 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 27 21:33:50.524734 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.524709 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 21:33:50.524734 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.524725 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 21:33:50.524786 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.524736 2556 kubelet.go:398] "Adding apiserver pod source" Jul 27 21:33:50.524786 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.524746 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 21:33:50.525177 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.525158 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:33:50.525238 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.525186 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 21:33:50.526615 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.526595 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g92vx" Jul 27 21:33:50.527740 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.527707 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 21:33:50.529052 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529038 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 21:33:50.529113 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529069 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 21:33:50.529607 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529595 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 21:33:50.529637 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529612 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 21:33:50.529637 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529619 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 21:33:50.529637 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529624 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 21:33:50.529637 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529632 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 21:33:50.529637 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529637 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 21:33:50.529785 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529696 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 21:33:50.529785 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529703 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 21:33:50.529785 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529711 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 21:33:50.529785 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529716 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 21:33:50.529785 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529725 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 21:33:50.529785 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.529733 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 21:33:50.531997 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.531986 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 21:33:50.531997 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.531996 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 21:33:50.534285 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.534264 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g92vx" Jul 27 21:33:50.534824 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.534797 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 21:33:50.535243 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.535217 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 21:33:50.535318 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.535217 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 21:33:50.536317 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.536175 2556 server.go:1268] "Started kubelet" Jul 27 21:33:50.536491 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.536415 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 21:33:50.536544 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.536518 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 21:33:50.536612 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.536537 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 21:33:50.537434 ip-10-0-133-37 systemd[1]: Started Kubernetes Kubelet. Jul 27 21:33:50.537989 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.537967 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 21:33:50.541344 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.541323 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 27 21:33:50.542592 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.542562 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 21:33:50.548508 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.548490 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 21:33:50.548508 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.548502 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 21:33:50.548634 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.548611 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:50.549038 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549015 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 21:33:50.549133 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549015 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 21:33:50.549133 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549060 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 21:33:50.549244 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549227 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 27 21:33:50.549297 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549246 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 27 21:33:50.549536 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549518 2556 factory.go:55] Registering systemd factory Jul 27 21:33:50.549682 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549669 2556 factory.go:223] Registration of the systemd container factory successfully Jul 27 21:33:50.549958 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549939 2556 factory.go:153] Registering CRI-O factory Jul 27 21:33:50.549958 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.549961 2556 factory.go:223] Registration of the crio container factory successfully Jul 27 21:33:50.550070 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.550014 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 21:33:50.550070 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.550037 2556 factory.go:103] Registering Raw factory Jul 27 21:33:50.550070 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.550055 2556 manager.go:1196] Started watching for new ooms in manager Jul 27 21:33:50.550777 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.550759 2556 manager.go:349] Starting recovery of all containers Jul 27 21:33:50.551082 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.551056 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:33:50.554556 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.554536 2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-37.ec2.internal\" not found" node="ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.561299 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.561282 2556 manager.go:354] Recovery completed Jul 27 21:33:50.565582 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.565564 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:33:50.567445 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.567428 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:33:50.567539 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.567462 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:33:50.567539 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.567473 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:33:50.568040 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.568025 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 21:33:50.568040 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.568036 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:33:50.568040 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.568040 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 21:33:50.568223 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.568060 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 27 21:33:50.568709 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.568694 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:33:50.568783 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.568726 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:33:50.568783 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.568742 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:33:50.569534 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.569521 2556 policy_none.go:49] "None policy: Start" Jul 27 21:33:50.569579 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.569537 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 21:33:50.569579 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.569548 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 21:33:50.570703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.570693 2556 policy_none.go:47] "Start" Jul 27 21:33:50.612663 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.612469 2556 manager.go:337] "Starting Device Plugin manager" Jul 27 21:33:50.612808 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.612697 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 21:33:50.612808 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.612711 2556 server.go:87] "Starting device plugin registration server" Jul 27 21:33:50.613005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.612990 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 21:33:50.613056 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.613006 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 21:33:50.613124 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.613107 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 21:33:50.613237 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.613200 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 21:33:50.613237 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.613214 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 21:33:50.613815 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.613792 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 21:33:50.613912 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.613867 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:50.714156 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.714072 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:33:50.715228 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.715205 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:33:50.715313 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.715237 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:33:50.715313 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.715252 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:33:50.715313 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.715297 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.725309 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.725287 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.725405 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.725317 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-37.ec2.internal\": node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:50.741588 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.741556 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:50.769548 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.769504 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 21:33:50.770928 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.770902 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 21:33:50.771036 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.770953 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 21:33:50.771036 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.770986 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 21:33:50.771036 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.771028 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 21:33:50.773373 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.773354 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:33:50.842643 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.842603 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:50.872002 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.871963 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal"] Jul 27 21:33:50.872085 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.872051 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:33:50.873084 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.873069 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:33:50.873154 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.873102 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:33:50.873154 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.873115 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:33:50.874371 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.874358 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:33:50.874498 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.874482 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.874552 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.874513 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:33:50.875129 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.875113 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:33:50.875234 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.875138 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:33:50.875234 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.875149 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:33:50.875234 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.875159 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:33:50.875234 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.875189 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:33:50.875234 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.875204 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:33:50.876391 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.876377 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.876444 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.876405 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 21:33:50.877470 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.877455 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientMemory" Jul 27 21:33:50.877566 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.877480 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 21:33:50.877566 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.877490 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeHasSufficientPID" Jul 27 21:33:50.899642 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.899613 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-37.ec2.internal\" not found" node="ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.904159 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.904138 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-37.ec2.internal\" not found" node="ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.942858 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:50.942818 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:50.950603 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.950580 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ba36726f14a0ca690cc34eac1afb86d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"6ba36726f14a0ca690cc34eac1afb86d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.950664 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.950610 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ba36726f14a0ca690cc34eac1afb86d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"6ba36726f14a0ca690cc34eac1afb86d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:50.950664 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:50.950632 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0768a6ccd589953a80216109a3092621-config\") pod \"kube-apiserver-proxy-ip-10-0-133-37.ec2.internal\" (UID: \"0768a6ccd589953a80216109a3092621\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Jul 27 21:33:51.043223 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:51.043196 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:51.051557 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:51.051532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ba36726f14a0ca690cc34eac1afb86d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"6ba36726f14a0ca690cc34eac1afb86d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:51.051664 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:51.051615 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ba36726f14a0ca690cc34eac1afb86d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"6ba36726f14a0ca690cc34eac1afb86d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:51.051664 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:51.051626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ba36726f14a0ca690cc34eac1afb86d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"6ba36726f14a0ca690cc34eac1afb86d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:51.051664 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:51.051656 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0768a6ccd589953a80216109a3092621-config\") pod \"kube-apiserver-proxy-ip-10-0-133-37.ec2.internal\" (UID: \"0768a6ccd589953a80216109a3092621\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Jul 27 21:33:51.051664 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:51.051626 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ba36726f14a0ca690cc34eac1afb86d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal\" (UID: \"6ba36726f14a0ca690cc34eac1afb86d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:51.051783 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:51.051692 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0768a6ccd589953a80216109a3092621-config\") pod \"kube-apiserver-proxy-ip-10-0-133-37.ec2.internal\" (UID: \"0768a6ccd589953a80216109a3092621\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Jul 27 21:33:52.558605 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:52.558566 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:52.617148 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:52.617106 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:52.622432 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:52.622415 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Jul 27 21:33:52.659461 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:52.659429 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:52.760043 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:52.759955 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:52.860503 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:52.860470 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:52.888953 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:52.888928 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 21:33:52.889380 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:52.889083 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 21:33:52.889380 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:52.889111 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 21:33:52.956728 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:52.956685 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 21:28:51 +0000 UTC" deadline="2028-03-31 05:57:04.721794583 +0000 UTC" Jul 27 21:33:52.956728 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:52.956721 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14696h23m11.765078528s" Jul 27 21:33:52.960554 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:52.960531 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:53.061622 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:53.061551 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:53.162203 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:53.162161 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:53.186988 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:53.186934 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba36726f14a0ca690cc34eac1afb86d.slice/crio-bbfca3e9301a2ae5d49f4e7cdcb16a0af3dbaa8eb54a4b80c2dc671fa4654d89 WatchSource:0}: Error finding container bbfca3e9301a2ae5d49f4e7cdcb16a0af3dbaa8eb54a4b80c2dc671fa4654d89: Status 404 returned error can't find the container with id bbfca3e9301a2ae5d49f4e7cdcb16a0af3dbaa8eb54a4b80c2dc671fa4654d89 Jul 27 21:33:53.187261 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:53.187240 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0768a6ccd589953a80216109a3092621.slice/crio-63289bb0b9e5f75efed88b731c13581c9b74ec45252868abadbbd90cad266244 WatchSource:0}: Error finding container 63289bb0b9e5f75efed88b731c13581c9b74ec45252868abadbbd90cad266244: Status 404 returned error can't find the container with id 63289bb0b9e5f75efed88b731c13581c9b74ec45252868abadbbd90cad266244 Jul 27 21:33:53.190720 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.190705 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:33:53.262950 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:53.262908 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:53.334283 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.334204 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:33:53.364067 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:53.364040 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-37.ec2.internal\" not found" Jul 27 21:33:53.412387 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.412358 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:33:53.463749 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.463712 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" Jul 27 21:33:53.475396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.475369 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:33:53.476378 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.476364 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" Jul 27 21:33:53.485191 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.485168 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 21:33:53.913620 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.913563 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:33:53.940909 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.940878 2556 apiserver.go:52] "Watching apiserver" Jul 27 21:33:53.950889 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.950860 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 21:33:53.952600 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.952570 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal","openshift-multus/network-metrics-daemon-x9ch7","openshift-network-diagnostics/network-check-target-85djc","openshift-network-operator/iptables-alerter-k9mbf","kube-system/global-pull-secret-syncer-m24q9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb","openshift-dns/node-resolver-qwnv7","openshift-image-registry/node-ca-7rkt6","openshift-multus/multus-additional-cni-plugins-bwp4t","openshift-multus/multus-cbc84","openshift-ovn-kubernetes/ovnkube-node-cks79","kube-system/konnectivity-agent-fmvdv","kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7jx27"] Jul 27 21:33:53.953910 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.953887 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 21:33:53.954843 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.954820 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:53.956224 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.956203 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.957554 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.957531 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 21:33:53.957652 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.957599 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bzc5h\"" Jul 27 21:33:53.957866 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.957850 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:53.957924 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.957865 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 21:33:53.957958 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:53.957927 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:33:53.958751 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.958731 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 21:33:53.958934 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.958920 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 21:33:53.959146 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.959130 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g5hgq\"" Jul 27 21:33:53.959272 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.959260 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 21:33:53.959419 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.959408 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 21:33:53.959523 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.959513 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 21:33:53.959806 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.959781 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 21:33:53.960126 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.960107 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:53.961950 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.961929 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:53.962048 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:53.962018 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:33:53.962420 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.962404 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 21:33:53.962561 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.962541 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:33:53.962635 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.962618 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-26t5b\"" Jul 27 21:33:53.963050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.963035 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 21:33:53.963502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.963483 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.963929 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.963911 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.965491 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.965821 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.966016 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.966272 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x6q7n\"" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.966524 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.967758 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.968415 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 21:33:53.968694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.968546 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fsgcq\"" Jul 27 21:33:53.969013 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.968726 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 21:33:53.969402 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.969096 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 21:33:53.969402 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.969119 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:53.969402 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:53.969194 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:33:53.969402 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.969303 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 21:33:53.969640 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.969536 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 21:33:53.970626 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.970046 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 21:33:53.970626 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.970098 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8c4jf\"" Jul 27 21:33:53.970626 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.970502 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:33:53.970626 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.970567 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbc84" Jul 27 21:33:53.974067 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.974036 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 21:33:53.974437 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.974418 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bpkp7\"" Jul 27 21:33:53.974663 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.974628 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:53.975764 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.975656 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976265 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-run-netns\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976281 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976291 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-node-log\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976338 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea75c0a0-edbc-4a2d-8993-f4377d799d19-iptables-alerter-script\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976367 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr427\" (UniqueName: \"kubernetes.io/projected/eeca7b48-3820-4595-88ee-37d4c98d392a-kube-api-access-qr427\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976392 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976417 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5f0961-958b-4d25-846a-6320dc6a87be-tmp-dir\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976485 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea75c0a0-edbc-4a2d-8993-f4377d799d19-host-slash\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8093b04e-e222-499b-b2e2-8bc7b0fc351d-kubelet-config\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976532 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-cni-netd\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976571 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovnkube-config\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976593 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-env-overrides\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976615 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-os-release\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976638 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976701 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a5f0961-958b-4d25-846a-6320dc6a87be-hosts-file\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:53.977286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976749 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8093b04e-e222-499b-b2e2-8bc7b0fc351d-dbus\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:53.978323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976780 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.978323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976848 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-device-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.978323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976875 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-slash\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.978323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976920 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:53.978323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.976931 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 21:33:53.982588 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982542 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.982733 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982622 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt622\" (UniqueName: \"kubernetes.io/projected/6d6b8115-32c2-440c-8b91-099f8716b64c-kube-api-access-tt622\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.982733 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982652 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-socket-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.982733 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982712 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-etc-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.982962 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982744 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-log-socket\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.982962 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982776 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-cni-bin\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.982962 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982810 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m5fqm\"" Jul 27 21:33:53.982962 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982813 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.982962 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.982962 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982959 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxk4\" (UniqueName: \"kubernetes.io/projected/ea75c0a0-edbc-4a2d-8993-f4377d799d19-kube-api-access-ftxk4\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.982978 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-registration-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983000 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-kubelet\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983033 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-var-lib-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjnh\" (UniqueName: \"kubernetes.io/projected/9a5f0961-958b-4d25-846a-6320dc6a87be-kube-api-access-gzjnh\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983070 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983091 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovnkube-script-lib\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983110 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrknt\" (UniqueName: \"kubernetes.io/projected/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-kube-api-access-rrknt\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983130 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-cnibin\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983161 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983183 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-systemd\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983203 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-ovn\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983228 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983249 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovn-node-metrics-cert\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983269 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-system-cni-dir\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983302 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983322 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-sys-fs\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:53.983396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983352 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-systemd-units\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:53.984186 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983514 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 21:33:53.984186 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983593 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zv6n8\"" Jul 27 21:33:53.984186 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983729 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 21:33:53.984186 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.983915 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 21:33:53.994318 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:53.994288 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnxv4" Jul 27 21:33:54.003876 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.003849 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnxv4" Jul 27 21:33:54.064536 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.064506 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 21:33:54.084143 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084116 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:54.084306 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-k8s-cni-cncf-io\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.084306 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084175 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-netns\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.084306 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-multus-certs\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.084306 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084270 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-socket-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084310 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-log-socket\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084341 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084372 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-daemon-config\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084399 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-etc-kubernetes\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084421 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-sys\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084423 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-socket-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084428 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-log-socket\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084445 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084464 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-lib-modules\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.084502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084500 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9vc\" (UniqueName: \"kubernetes.io/projected/ae9892bc-23d3-4919-851b-5a2b535e58f8-kube-api-access-tz9vc\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084531 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxk4\" (UniqueName: \"kubernetes.io/projected/ea75c0a0-edbc-4a2d-8993-f4377d799d19-kube-api-access-ftxk4\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084555 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-kubelet\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084578 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-var-lib-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084600 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjnh\" (UniqueName: \"kubernetes.io/projected/9a5f0961-958b-4d25-846a-6320dc6a87be-kube-api-access-gzjnh\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084618 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084627 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-var-lib-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084633 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovnkube-script-lib\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084648 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrknt\" (UniqueName: \"kubernetes.io/projected/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-kube-api-access-rrknt\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084657 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-kubelet\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084686 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-kubernetes\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084706 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysctl-d\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-systemd\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084740 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-ovn\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.084749 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084760 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-system-cni-dir\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084785 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d516a54d-e9c9-4a25-9e86-b2b1503eb935-cni-binary-copy\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.084927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084808 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jz4q\" (UniqueName: \"kubernetes.io/projected/edb9df30-5601-4bcc-9d00-3d531da72dd5-kube-api-access-9jz4q\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.084847 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret podName:8093b04e-e222-499b-b2e2-8bc7b0fc351d nodeName:}" failed. No retries permitted until 2026-07-27 21:33:54.584803953 +0000 UTC m=+2.993225990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret") pod "global-pull-secret-syncer-m24q9" (UID: "8093b04e-e222-499b-b2e2-8bc7b0fc351d") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084872 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084892 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-sys-fs\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-systemd-units\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084933 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-run-netns\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084949 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-var-lib-kubelet\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084966 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea75c0a0-edbc-4a2d-8993-f4377d799d19-iptables-alerter-script\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.084980 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-cni-bin\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085028 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb9df30-5601-4bcc-9d00-3d531da72dd5-serviceca\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085029 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-systemd-units\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085140 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysconfig\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085149 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-sys-fs\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085172 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5f0961-958b-4d25-846a-6320dc6a87be-tmp-dir\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085179 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-system-cni-dir\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085204 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-run-netns\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.085653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085210 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-etc-selinux\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085222 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea75c0a0-edbc-4a2d-8993-f4377d799d19-host-slash\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085228 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-ovn\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085251 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8093b04e-e222-499b-b2e2-8bc7b0fc351d-kubelet-config\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085249 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-systemd\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085284 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-env-overrides\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085309 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea75c0a0-edbc-4a2d-8993-f4377d799d19-host-slash\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085309 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085370 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8093b04e-e222-499b-b2e2-8bc7b0fc351d-kubelet-config\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085392 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-conf-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a5f0961-958b-4d25-846a-6320dc6a87be-hosts-file\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085438 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovnkube-script-lib\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8093b04e-e222-499b-b2e2-8bc7b0fc351d-dbus\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085470 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085506 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9a5f0961-958b-4d25-846a-6320dc6a87be-hosts-file\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-host\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.086176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085538 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5f0961-958b-4d25-846a-6320dc6a87be-tmp-dir\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7347070d-ff62-4d14-9b12-433decb4d5b6-agent-certs\") pod \"konnectivity-agent-fmvdv\" (UID: \"7347070d-ff62-4d14-9b12-433decb4d5b6\") " pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085572 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea75c0a0-edbc-4a2d-8993-f4377d799d19-iptables-alerter-script\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085582 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-etc-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085607 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-etc-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085626 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-cni-bin\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085702 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-env-overrides\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085694 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-cni-bin\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085661 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8093b04e-e222-499b-b2e2-8bc7b0fc351d-dbus\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085757 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6z6h\" (UniqueName: \"kubernetes.io/projected/ac1951dc-f8fe-4888-81ff-71d3dd80d587-kube-api-access-p6z6h\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085773 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-os-release\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085791 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-cni-multus\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085806 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-kubelet\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085821 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-hostroot\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085840 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-registration-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085855 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:54.086631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-system-cni-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085885 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-modprobe-d\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085899 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-systemd\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-tuned\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-cnibin\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085944 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085960 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovn-node-metrics-cert\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysctl-conf\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.085992 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-node-log\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086036 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-cnibin\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086054 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr427\" (UniqueName: \"kubernetes.io/projected/eeca7b48-3820-4595-88ee-37d4c98d392a-kube-api-access-qr427\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086071 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086071 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086089 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086115 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c29kh\" (UniqueName: \"kubernetes.io/projected/d516a54d-e9c9-4a25-9e86-b2b1503eb935-kube-api-access-c29kh\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086145 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7347070d-ff62-4d14-9b12-433decb4d5b6-konnectivity-ca\") pod \"konnectivity-agent-fmvdv\" (UID: \"7347070d-ff62-4d14-9b12-433decb4d5b6\") " pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-cnibin\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.087208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086150 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086173 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-cni-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086194 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-node-log\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086207 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-cni-netd\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086226 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovnkube-config\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086241 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-os-release\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086247 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-run-openvswitch\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086261 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt622\" (UniqueName: \"kubernetes.io/projected/6d6b8115-32c2-440c-8b91-099f8716b64c-kube-api-access-tt622\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086281 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb9df30-5601-4bcc-9d00-3d531da72dd5-host\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086302 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-run\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086319 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9892bc-23d3-4919-851b-5a2b535e58f8-tmp\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086334 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-socket-dir-parent\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086376 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086394 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-device-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086410 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-slash\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086462 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-slash\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086499 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.087791 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086513 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 21:33:54.088300 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086537 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-device-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.088300 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086568 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-cni-netd\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.088300 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086639 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d6b8115-32c2-440c-8b91-099f8716b64c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.088300 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086694 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eeca7b48-3820-4595-88ee-37d4c98d392a-registration-dir\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.088300 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086718 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.088300 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.086829 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6d6b8115-32c2-440c-8b91-099f8716b64c-os-release\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.088300 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.087435 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovnkube-config\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.089969 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.089947 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-ovn-node-metrics-cert\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.094369 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.094342 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:33:54.094369 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.094360 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:33:54.094369 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.094373 2556 projected.go:196] Error preparing data for projected volume kube-api-access-t9cvj for pod openshift-network-diagnostics/network-check-target-85djc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:54.094582 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.094428 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj podName:af0b01fc-decc-4da8-9310-445e5e4e3b89 nodeName:}" failed. No retries permitted until 2026-07-27 21:33:54.594413363 +0000 UTC m=+3.002835394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t9cvj" (UniqueName: "kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj") pod "network-check-target-85djc" (UID: "af0b01fc-decc-4da8-9310-445e5e4e3b89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:54.097535 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.097004 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjnh\" (UniqueName: \"kubernetes.io/projected/9a5f0961-958b-4d25-846a-6320dc6a87be-kube-api-access-gzjnh\") pod \"node-resolver-qwnv7\" (UID: \"9a5f0961-958b-4d25-846a-6320dc6a87be\") " pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:54.097535 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.097169 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxk4\" (UniqueName: \"kubernetes.io/projected/ea75c0a0-edbc-4a2d-8993-f4377d799d19-kube-api-access-ftxk4\") pod \"iptables-alerter-k9mbf\" (UID: \"ea75c0a0-edbc-4a2d-8993-f4377d799d19\") " pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:54.097535 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.097510 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrknt\" (UniqueName: \"kubernetes.io/projected/ae79c26d-4dbc-4d9a-aa29-9a07273245c0-kube-api-access-rrknt\") pod \"ovnkube-node-cks79\" (UID: \"ae79c26d-4dbc-4d9a-aa29-9a07273245c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.102770 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.102725 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr427\" (UniqueName: \"kubernetes.io/projected/eeca7b48-3820-4595-88ee-37d4c98d392a-kube-api-access-qr427\") pod \"aws-ebs-csi-driver-node-zsccb\" (UID: \"eeca7b48-3820-4595-88ee-37d4c98d392a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.103495 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.103478 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt622\" (UniqueName: \"kubernetes.io/projected/6d6b8115-32c2-440c-8b91-099f8716b64c-kube-api-access-tt622\") pod \"multus-additional-cni-plugins-bwp4t\" (UID: \"6d6b8115-32c2-440c-8b91-099f8716b64c\") " pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.186738 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysctl-conf\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.186738 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186691 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-cnibin\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186766 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-cnibin\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186801 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c29kh\" (UniqueName: \"kubernetes.io/projected/d516a54d-e9c9-4a25-9e86-b2b1503eb935-kube-api-access-c29kh\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7347070d-ff62-4d14-9b12-433decb4d5b6-konnectivity-ca\") pod \"konnectivity-agent-fmvdv\" (UID: \"7347070d-ff62-4d14-9b12-433decb4d5b6\") " pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186840 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-cni-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186840 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysctl-conf\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186857 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb9df30-5601-4bcc-9d00-3d531da72dd5-host\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186871 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-run\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9892bc-23d3-4919-851b-5a2b535e58f8-tmp\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.186917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186900 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-socket-dir-parent\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186927 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-k8s-cni-cncf-io\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186943 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-netns\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186958 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-multus-certs\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186975 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-daemon-config\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.186990 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-etc-kubernetes\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187006 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-sys\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187020 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-lib-modules\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187036 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9vc\" (UniqueName: \"kubernetes.io/projected/ae9892bc-23d3-4919-851b-5a2b535e58f8-kube-api-access-tz9vc\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-kubernetes\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187078 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysctl-d\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187090 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-cni-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187094 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d516a54d-e9c9-4a25-9e86-b2b1503eb935-cni-binary-copy\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187128 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jz4q\" (UniqueName: \"kubernetes.io/projected/edb9df30-5601-4bcc-9d00-3d531da72dd5-kube-api-access-9jz4q\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-var-lib-kubelet\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187167 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-cni-bin\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187182 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb9df30-5601-4bcc-9d00-3d531da72dd5-serviceca\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187197 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysconfig\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187216 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-conf-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187237 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-host\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187252 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7347070d-ff62-4d14-9b12-433decb4d5b6-agent-certs\") pod \"konnectivity-agent-fmvdv\" (UID: \"7347070d-ff62-4d14-9b12-433decb4d5b6\") " pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187270 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6z6h\" (UniqueName: \"kubernetes.io/projected/ac1951dc-f8fe-4888-81ff-71d3dd80d587-kube-api-access-p6z6h\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-os-release\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187301 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-cni-multus\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187323 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-var-lib-kubelet\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187360 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-cni-bin\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187378 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-kubelet\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187432 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edb9df30-5601-4bcc-9d00-3d531da72dd5-host\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.187574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187477 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-run\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187662 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d516a54d-e9c9-4a25-9e86-b2b1503eb935-cni-binary-copy\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187695 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edb9df30-5601-4bcc-9d00-3d531da72dd5-serviceca\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187744 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-netns\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysconfig\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187789 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-multus-certs\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187795 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-conf-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187858 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-sys\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187926 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-run-k8s-cni-cncf-io\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187955 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7347070d-ff62-4d14-9b12-433decb4d5b6-konnectivity-ca\") pod \"konnectivity-agent-fmvdv\" (UID: \"7347070d-ff62-4d14-9b12-433decb4d5b6\") " pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:54.188030 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187984 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-lib-modules\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188344 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188035 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-kubernetes\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188344 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188141 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-sysctl-d\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188344 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188224 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-daemon-config\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188344 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188281 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-etc-kubernetes\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188344 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188297 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-host\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188570 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188352 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-os-release\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188570 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188398 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-cni-multus\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188712 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188690 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-multus-socket-dir-parent\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188712 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.187334 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-host-var-lib-kubelet\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188834 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188767 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-hostroot\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188834 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188799 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:54.188941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188843 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-system-cni-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188870 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-modprobe-d\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188886 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-hostroot\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.188941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-systemd\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.188941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188929 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-tuned\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.189164 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188943 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d516a54d-e9c9-4a25-9e86-b2b1503eb935-system-cni-dir\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.189164 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.188931 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:54.189164 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.188979 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-systemd\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.189164 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.189034 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs podName:ac1951dc-f8fe-4888-81ff-71d3dd80d587 nodeName:}" failed. No retries permitted until 2026-07-27 21:33:54.689019443 +0000 UTC m=+3.097441450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs") pod "network-metrics-daemon-x9ch7" (UID: "ac1951dc-f8fe-4888-81ff-71d3dd80d587") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:54.189164 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.189064 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-modprobe-d\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.190775 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.190739 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae9892bc-23d3-4919-851b-5a2b535e58f8-tmp\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.191347 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.191326 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ae9892bc-23d3-4919-851b-5a2b535e58f8-etc-tuned\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.191458 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.191384 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7347070d-ff62-4d14-9b12-433decb4d5b6-agent-certs\") pod \"konnectivity-agent-fmvdv\" (UID: \"7347070d-ff62-4d14-9b12-433decb4d5b6\") " pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:54.191571 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.191511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" event={"ID":"6ba36726f14a0ca690cc34eac1afb86d","Type":"ContainerStarted","Data":"bbfca3e9301a2ae5d49f4e7cdcb16a0af3dbaa8eb54a4b80c2dc671fa4654d89"} Jul 27 21:33:54.192687 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.192647 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" event={"ID":"0768a6ccd589953a80216109a3092621","Type":"ContainerStarted","Data":"63289bb0b9e5f75efed88b731c13581c9b74ec45252868abadbbd90cad266244"} Jul 27 21:33:54.195623 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.195562 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c29kh\" (UniqueName: \"kubernetes.io/projected/d516a54d-e9c9-4a25-9e86-b2b1503eb935-kube-api-access-c29kh\") pod \"multus-cbc84\" (UID: \"d516a54d-e9c9-4a25-9e86-b2b1503eb935\") " pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.196105 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.196088 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6z6h\" (UniqueName: \"kubernetes.io/projected/ac1951dc-f8fe-4888-81ff-71d3dd80d587-kube-api-access-p6z6h\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:54.196614 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.196585 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9vc\" (UniqueName: \"kubernetes.io/projected/ae9892bc-23d3-4919-851b-5a2b535e58f8-kube-api-access-tz9vc\") pod \"tuned-7jx27\" (UID: \"ae9892bc-23d3-4919-851b-5a2b535e58f8\") " pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.196936 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.196918 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jz4q\" (UniqueName: \"kubernetes.io/projected/edb9df30-5601-4bcc-9d00-3d531da72dd5-kube-api-access-9jz4q\") pod \"node-ca-7rkt6\" (UID: \"edb9df30-5601-4bcc-9d00-3d531da72dd5\") " pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.270117 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.270074 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qwnv7" Jul 27 21:33:54.276591 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.276567 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:33:54.293586 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.293561 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9mbf" Jul 27 21:33:54.300558 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.300531 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" Jul 27 21:33:54.307382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.307353 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" Jul 27 21:33:54.316165 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.316138 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7jx27" Jul 27 21:33:54.323326 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.323300 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cbc84" Jul 27 21:33:54.331129 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.331097 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rkt6" Jul 27 21:33:54.338074 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.338042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:33:54.591752 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.591719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:54.591922 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.591847 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:54.591922 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.591915 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret podName:8093b04e-e222-499b-b2e2-8bc7b0fc351d nodeName:}" failed. No retries permitted until 2026-07-27 21:33:55.591894227 +0000 UTC m=+4.000316236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret") pod "global-pull-secret-syncer-m24q9" (UID: "8093b04e-e222-499b-b2e2-8bc7b0fc351d") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:54.692159 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.692131 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:54.692326 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:54.692187 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:54.692326 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.692282 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:54.692326 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.692298 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:33:54.692326 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.692318 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:33:54.692326 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.692327 2556 projected.go:196] Error preparing data for projected volume kube-api-access-t9cvj for pod openshift-network-diagnostics/network-check-target-85djc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:54.692506 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.692339 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs podName:ac1951dc-f8fe-4888-81ff-71d3dd80d587 nodeName:}" failed. No retries permitted until 2026-07-27 21:33:55.69232303 +0000 UTC m=+4.100745038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs") pod "network-metrics-daemon-x9ch7" (UID: "ac1951dc-f8fe-4888-81ff-71d3dd80d587") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:54.692506 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:54.692370 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj podName:af0b01fc-decc-4da8-9310-445e5e4e3b89 nodeName:}" failed. No retries permitted until 2026-07-27 21:33:55.692356734 +0000 UTC m=+4.100778757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9cvj" (UniqueName: "kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj") pod "network-check-target-85djc" (UID: "af0b01fc-decc-4da8-9310-445e5e4e3b89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:55.006471 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.006437 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:28:53 +0000 UTC" deadline="2028-03-13 04:22:09.727887332 +0000 UTC" Jul 27 21:33:55.006471 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.006466 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14262h48m14.721424393s" Jul 27 21:33:55.016560 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:55.016538 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9892bc_23d3_4919_851b_5a2b535e58f8.slice/crio-8e084e82b2e2c9c8ab25c3ae30638c5dec18403a44e1396c7c48dd0768c889cb WatchSource:0}: Error finding container 8e084e82b2e2c9c8ab25c3ae30638c5dec18403a44e1396c7c48dd0768c889cb: Status 404 returned error can't find the container with id 8e084e82b2e2c9c8ab25c3ae30638c5dec18403a44e1396c7c48dd0768c889cb Jul 27 21:33:55.027853 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:55.027830 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7347070d_ff62_4d14_9b12_433decb4d5b6.slice/crio-226a5c3707a9de3c4b211eb45f4d39c3943372054b3e79ab309f3f2d65be0d68 WatchSource:0}: Error finding container 226a5c3707a9de3c4b211eb45f4d39c3943372054b3e79ab309f3f2d65be0d68: Status 404 returned error can't find the container with id 226a5c3707a9de3c4b211eb45f4d39c3943372054b3e79ab309f3f2d65be0d68 Jul 27 21:33:55.032814 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:55.031954 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb9df30_5601_4bcc_9d00_3d531da72dd5.slice/crio-de21201c74b3434021b065aa77baba7fb54b7a9058fee5fe39c3a179526e82fd WatchSource:0}: Error finding container de21201c74b3434021b065aa77baba7fb54b7a9058fee5fe39c3a179526e82fd: Status 404 returned error can't find the container with id de21201c74b3434021b065aa77baba7fb54b7a9058fee5fe39c3a179526e82fd Jul 27 21:33:55.033979 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:55.033953 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd516a54d_e9c9_4a25_9e86_b2b1503eb935.slice/crio-6c6e5952bd2519784d42f7050505d6956a1c9ec63fe59fe9d2693a0d36bef247 WatchSource:0}: Error finding container 6c6e5952bd2519784d42f7050505d6956a1c9ec63fe59fe9d2693a0d36bef247: Status 404 returned error can't find the container with id 6c6e5952bd2519784d42f7050505d6956a1c9ec63fe59fe9d2693a0d36bef247 Jul 27 21:33:55.035297 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:55.035221 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6b8115_32c2_440c_8b91_099f8716b64c.slice/crio-f172c5dac0737aa485399f152634528c2a6820ce1e0287f2e05ec3b998fa960a WatchSource:0}: Error finding container f172c5dac0737aa485399f152634528c2a6820ce1e0287f2e05ec3b998fa960a: Status 404 returned error can't find the container with id f172c5dac0737aa485399f152634528c2a6820ce1e0287f2e05ec3b998fa960a Jul 27 21:33:55.037552 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:33:55.037528 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeca7b48_3820_4595_88ee_37d4c98d392a.slice/crio-d639e0713d82bb9a307746130634cfee53f59f3deac0c8c52eaf4f9a09aa7ad0 WatchSource:0}: Error finding container d639e0713d82bb9a307746130634cfee53f59f3deac0c8c52eaf4f9a09aa7ad0: Status 404 returned error can't find the container with id d639e0713d82bb9a307746130634cfee53f59f3deac0c8c52eaf4f9a09aa7ad0 Jul 27 21:33:55.186514 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.186320 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:55.186703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.186366 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:55.186703 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.186639 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:33:55.186703 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.186663 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:33:55.195020 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.194988 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jx27" event={"ID":"ae9892bc-23d3-4919-851b-5a2b535e58f8","Type":"ContainerStarted","Data":"8e084e82b2e2c9c8ab25c3ae30638c5dec18403a44e1396c7c48dd0768c889cb"} Jul 27 21:33:55.196733 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.196703 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" event={"ID":"0768a6ccd589953a80216109a3092621","Type":"ContainerStarted","Data":"8ce6fe8d389830fc09c6fe06ae7d9c1f3ff299bfd14297f11012c238c4b247f6"} Jul 27 21:33:55.197615 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.197596 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" event={"ID":"eeca7b48-3820-4595-88ee-37d4c98d392a","Type":"ContainerStarted","Data":"d639e0713d82bb9a307746130634cfee53f59f3deac0c8c52eaf4f9a09aa7ad0"} Jul 27 21:33:55.198559 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.198542 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rkt6" event={"ID":"edb9df30-5601-4bcc-9d00-3d531da72dd5","Type":"ContainerStarted","Data":"de21201c74b3434021b065aa77baba7fb54b7a9058fee5fe39c3a179526e82fd"} Jul 27 21:33:55.199464 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.199444 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmvdv" event={"ID":"7347070d-ff62-4d14-9b12-433decb4d5b6","Type":"ContainerStarted","Data":"226a5c3707a9de3c4b211eb45f4d39c3943372054b3e79ab309f3f2d65be0d68"} Jul 27 21:33:55.200515 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.200490 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qwnv7" event={"ID":"9a5f0961-958b-4d25-846a-6320dc6a87be","Type":"ContainerStarted","Data":"4da202186450ee75e346fe6d8dd16a887a421688c20230a3cc42dd18b375e53a"} Jul 27 21:33:55.204876 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.204853 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9mbf" event={"ID":"ea75c0a0-edbc-4a2d-8993-f4377d799d19","Type":"ContainerStarted","Data":"47e81d23ed2956942625202c9ccf0eff01367e7cc9dae8732514cb940c1aa6dc"} Jul 27 21:33:55.205918 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.205890 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerStarted","Data":"f172c5dac0737aa485399f152634528c2a6820ce1e0287f2e05ec3b998fa960a"} Jul 27 21:33:55.206974 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.206954 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbc84" event={"ID":"d516a54d-e9c9-4a25-9e86-b2b1503eb935","Type":"ContainerStarted","Data":"6c6e5952bd2519784d42f7050505d6956a1c9ec63fe59fe9d2693a0d36bef247"} Jul 27 21:33:55.207898 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.207874 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"40ff52874d098462dffdc7388af30dd590c9f91392697f26e48ace1881778b1b"} Jul 27 21:33:55.214388 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.214349 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-37.ec2.internal" podStartSLOduration=2.214338888 podStartE2EDuration="2.214338888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:33:55.214167933 +0000 UTC m=+3.622589964" watchObservedRunningTime="2026-07-27 21:33:55.214338888 +0000 UTC m=+3.622760918" Jul 27 21:33:55.599181 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.599090 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:55.599340 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.599253 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:55.599340 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.599319 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret podName:8093b04e-e222-499b-b2e2-8bc7b0fc351d nodeName:}" failed. No retries permitted until 2026-07-27 21:33:57.599299639 +0000 UTC m=+6.007721648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret") pod "global-pull-secret-syncer-m24q9" (UID: "8093b04e-e222-499b-b2e2-8bc7b0fc351d") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:55.701168 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.700450 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:55.701168 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:55.700522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:55.701168 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.700777 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:55.701168 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.700842 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs podName:ac1951dc-f8fe-4888-81ff-71d3dd80d587 nodeName:}" failed. No retries permitted until 2026-07-27 21:33:57.700822321 +0000 UTC m=+6.109244332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs") pod "network-metrics-daemon-x9ch7" (UID: "ac1951dc-f8fe-4888-81ff-71d3dd80d587") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:55.701713 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.701550 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:33:55.701713 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.701580 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:33:55.701713 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.701602 2556 projected.go:196] Error preparing data for projected volume kube-api-access-t9cvj for pod openshift-network-diagnostics/network-check-target-85djc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:55.701713 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:55.701657 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj podName:af0b01fc-decc-4da8-9310-445e5e4e3b89 nodeName:}" failed. No retries permitted until 2026-07-27 21:33:57.701641451 +0000 UTC m=+6.110063477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9cvj" (UniqueName: "kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj") pod "network-check-target-85djc" (UID: "af0b01fc-decc-4da8-9310-445e5e4e3b89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:56.007082 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:56.007010 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:28:53 +0000 UTC" deadline="2028-03-31 16:09:59.749978697 +0000 UTC" Jul 27 21:33:56.007082 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:56.007075 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14706h36m3.74290778s" Jul 27 21:33:56.187642 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:56.187096 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:56.187642 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:56.187254 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:33:56.218260 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:56.218223 2556 generic.go:356] "Generic (PLEG): container finished" podID="6ba36726f14a0ca690cc34eac1afb86d" containerID="7d1a17731d6f65a96f95ffe6f1714f9c488ef53614d3af3034fa883a1b52ab5a" exitCode=0 Jul 27 21:33:56.218428 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:56.218305 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" event={"ID":"6ba36726f14a0ca690cc34eac1afb86d","Type":"ContainerDied","Data":"7d1a17731d6f65a96f95ffe6f1714f9c488ef53614d3af3034fa883a1b52ab5a"} Jul 27 21:33:57.186463 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:57.186411 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:57.186983 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.186564 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:33:57.186983 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:57.186762 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:57.186983 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.186853 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:33:57.250866 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:57.250782 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" event={"ID":"6ba36726f14a0ca690cc34eac1afb86d","Type":"ContainerStarted","Data":"d574c8154ac90db7c58c31fc2b4d94d40f207cd8b91df71acf9fe4a175049efc"} Jul 27 21:33:57.618574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:57.618532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:57.618797 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.618739 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:57.618868 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.618805 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret podName:8093b04e-e222-499b-b2e2-8bc7b0fc351d nodeName:}" failed. No retries permitted until 2026-07-27 21:34:01.618786651 +0000 UTC m=+10.027208664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret") pod "global-pull-secret-syncer-m24q9" (UID: "8093b04e-e222-499b-b2e2-8bc7b0fc351d") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:33:57.719894 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:57.719853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:57.720082 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:57.719950 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:57.720082 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.720076 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:57.720197 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.720142 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs podName:ac1951dc-f8fe-4888-81ff-71d3dd80d587 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:01.720121912 +0000 UTC m=+10.128543936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs") pod "network-metrics-daemon-x9ch7" (UID: "ac1951dc-f8fe-4888-81ff-71d3dd80d587") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:33:57.720631 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.720594 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:33:57.720631 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.720622 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:33:57.720631 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.720635 2556 projected.go:196] Error preparing data for projected volume kube-api-access-t9cvj for pod openshift-network-diagnostics/network-check-target-85djc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:57.720860 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:57.720701 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj podName:af0b01fc-decc-4da8-9310-445e5e4e3b89 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:01.720684608 +0000 UTC m=+10.129106629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9cvj" (UniqueName: "kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj") pod "network-check-target-85djc" (UID: "af0b01fc-decc-4da8-9310-445e5e4e3b89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:33:58.186922 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:58.186860 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:33:58.187335 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:58.186994 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:33:59.186439 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:59.186402 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:33:59.186652 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:59.186537 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:33:59.186991 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:33:59.186970 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:33:59.187379 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:33:59.187068 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:00.186580 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:00.186110 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:00.186580 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:00.186227 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:01.187515 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:01.186913 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:01.187515 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:01.186930 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:01.187515 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.187097 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:01.187515 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.187471 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:01.654749 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:01.654707 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:01.654952 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.654886 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:34:01.655017 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.654961 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret podName:8093b04e-e222-499b-b2e2-8bc7b0fc351d nodeName:}" failed. No retries permitted until 2026-07-27 21:34:09.654939511 +0000 UTC m=+18.063361533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret") pod "global-pull-secret-syncer-m24q9" (UID: "8093b04e-e222-499b-b2e2-8bc7b0fc351d") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:34:01.755514 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:01.755475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:01.755691 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:01.755547 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:01.755767 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.755729 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:34:01.755767 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.755751 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:34:01.755767 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.755764 2556 projected.go:196] Error preparing data for projected volume kube-api-access-t9cvj for pod openshift-network-diagnostics/network-check-target-85djc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:34:01.755914 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.755825 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj podName:af0b01fc-decc-4da8-9310-445e5e4e3b89 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:09.75580708 +0000 UTC m=+18.164229091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9cvj" (UniqueName: "kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj") pod "network-check-target-85djc" (UID: "af0b01fc-decc-4da8-9310-445e5e4e3b89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:34:01.756293 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.756207 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:34:01.756293 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:01.756263 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs podName:ac1951dc-f8fe-4888-81ff-71d3dd80d587 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:09.756247327 +0000 UTC m=+18.164669341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs") pod "network-metrics-daemon-x9ch7" (UID: "ac1951dc-f8fe-4888-81ff-71d3dd80d587") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:34:02.187658 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:02.187623 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:02.188028 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:02.187731 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:03.186413 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:03.186373 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:03.186599 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:03.186510 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:03.186599 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:03.186587 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:03.186832 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:03.186659 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:04.187726 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:04.187245 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:04.187726 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:04.187369 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:05.186757 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:05.186711 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:05.186757 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:05.186761 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:05.186973 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:05.186853 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:05.187035 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:05.186991 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:06.186359 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:06.186281 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:06.186799 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:06.186419 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:07.187036 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:07.186994 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:07.187490 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:07.187042 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:07.187490 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:07.187151 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:07.187490 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:07.187287 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:08.186310 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:08.186273 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:08.186471 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:08.186404 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:09.186223 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:09.186189 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:09.186771 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:09.186238 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:09.186771 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.186313 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:09.186771 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.186475 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:09.714461 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:09.714421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:09.714720 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.714530 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:34:09.714720 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.714591 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret podName:8093b04e-e222-499b-b2e2-8bc7b0fc351d nodeName:}" failed. No retries permitted until 2026-07-27 21:34:25.714575689 +0000 UTC m=+34.122997699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret") pod "global-pull-secret-syncer-m24q9" (UID: "8093b04e-e222-499b-b2e2-8bc7b0fc351d") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:34:09.815219 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:09.815184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:09.815396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:09.815229 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:09.815396 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.815326 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:34:09.815396 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.815358 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:34:09.815396 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.815373 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:34:09.815396 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.815383 2556 projected.go:196] Error preparing data for projected volume kube-api-access-t9cvj for pod openshift-network-diagnostics/network-check-target-85djc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:34:09.815396 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.815391 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs podName:ac1951dc-f8fe-4888-81ff-71d3dd80d587 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:25.815374748 +0000 UTC m=+34.223796756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs") pod "network-metrics-daemon-x9ch7" (UID: "ac1951dc-f8fe-4888-81ff-71d3dd80d587") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:34:09.815597 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:09.815425 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj podName:af0b01fc-decc-4da8-9310-445e5e4e3b89 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:25.815411011 +0000 UTC m=+34.223833036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9cvj" (UniqueName: "kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj") pod "network-check-target-85djc" (UID: "af0b01fc-decc-4da8-9310-445e5e4e3b89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:34:10.186491 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:10.186402 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:10.186969 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:10.186536 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:11.186994 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:11.186952 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:11.187486 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:11.187010 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:11.187486 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:11.187103 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:11.187486 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:11.187238 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:12.190074 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:12.188570 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:12.190074 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:12.188740 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:13.186368 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:13.186332 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:13.186575 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:13.186332 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:13.186575 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:13.186456 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:13.186575 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:13.186561 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:14.186467 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.186041 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:14.187135 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:14.186601 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:14.289892 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.289852 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7jx27" event={"ID":"ae9892bc-23d3-4919-851b-5a2b535e58f8","Type":"ContainerStarted","Data":"d5ce44aa75403fbc5842ea06500cf90ba95584bf126446460cc22890eb0004db"} Jul 27 21:34:14.291574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.291550 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" event={"ID":"eeca7b48-3820-4595-88ee-37d4c98d392a","Type":"ContainerStarted","Data":"2b5e18b797166c3cd8b5ff80a3e96ed18a0c7a82ff3047c02de674ed7c92db2b"} Jul 27 21:34:14.293248 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.293202 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rkt6" event={"ID":"edb9df30-5601-4bcc-9d00-3d531da72dd5","Type":"ContainerStarted","Data":"b247e29286f8358b725c9b53860276a26053fa705454fb8c06b75717cba931e6"} Jul 27 21:34:14.294821 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.294796 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmvdv" event={"ID":"7347070d-ff62-4d14-9b12-433decb4d5b6","Type":"ContainerStarted","Data":"ce88cca82446e8ca16bddc0066b511048663dd8b5d14fec329c6fc393d4f1bc7"} Jul 27 21:34:14.296175 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.296152 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qwnv7" event={"ID":"9a5f0961-958b-4d25-846a-6320dc6a87be","Type":"ContainerStarted","Data":"a2f34ffeba54c3fc165093c8096034f0b7ccc3daa6f7555a2a9b38cc992d5e71"} Jul 27 21:34:14.297493 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.297473 2556 generic.go:356] "Generic (PLEG): container finished" podID="6d6b8115-32c2-440c-8b91-099f8716b64c" containerID="575eb99c8badb2fb0e7b086436418766dd6313625b3f28563c3acf3da656d027" exitCode=0 Jul 27 21:34:14.297575 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.297552 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerDied","Data":"575eb99c8badb2fb0e7b086436418766dd6313625b3f28563c3acf3da656d027"} Jul 27 21:34:14.298920 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.298904 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cbc84" event={"ID":"d516a54d-e9c9-4a25-9e86-b2b1503eb935","Type":"ContainerStarted","Data":"2ab96d8b4efa263c7f614d9183435d65a5a155dcfc8346b2cb70c1525ca860f8"} Jul 27 21:34:14.300830 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.300811 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"b9bc78087fb0a01ca40b27658d8c35f61609d5b93bee8c51de17ba9c19732470"} Jul 27 21:34:14.300920 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.300881 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"a194caece5037544b86ea13914cb9077d19a5a69a4178a218b9e533b51999cfe"} Jul 27 21:34:14.300981 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.300931 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"cf17c8c2d25262d6e1b75499458b0eeebd19c5f48ac4c93dda5af0573300e609"} Jul 27 21:34:14.307741 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.307702 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7jx27" podStartSLOduration=3.487819875 podStartE2EDuration="22.307665555s" totalImagesPullingTime="18.81984568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:14.307360813 +0000 UTC m=+22.715782843" watchObservedRunningTime="2026-07-27 21:34:14.307665555 +0000 UTC m=+22.716111394" Jul 27 21:34:14.307859 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.307820 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-37.ec2.internal" podStartSLOduration=21.307811272 podStartE2EDuration="21.307811272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:33:57.26794236 +0000 UTC m=+5.676364391" watchObservedRunningTime="2026-07-27 21:34:14.307811272 +0000 UTC m=+22.716233304" Jul 27 21:34:14.348249 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.348155 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cbc84" podStartSLOduration=3.499365877 podStartE2EDuration="22.348140467s" totalImagesPullingTime="18.84877459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:14.347346795 +0000 UTC m=+22.755768823" watchObservedRunningTime="2026-07-27 21:34:14.348140467 +0000 UTC m=+22.756562497" Jul 27 21:34:14.381902 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.381853 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7rkt6" podStartSLOduration=3.579395746 podStartE2EDuration="22.381836183s" totalImagesPullingTime="18.802440437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:14.367150623 +0000 UTC m=+22.775572653" watchObservedRunningTime="2026-07-27 21:34:14.381836183 +0000 UTC m=+22.790258207" Jul 27 21:34:14.382246 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.382221 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fmvdv" podStartSLOduration=11.462617958 podStartE2EDuration="22.382211979s" totalImagesPullingTime="10.919594021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:14.381967277 +0000 UTC m=+22.790389289" watchObservedRunningTime="2026-07-27 21:34:14.382211979 +0000 UTC m=+22.790634009" Jul 27 21:34:14.396656 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:14.396615 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qwnv7" podStartSLOduration=3.586150265 podStartE2EDuration="22.396602924s" totalImagesPullingTime="18.810452659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:14.396560756 +0000 UTC m=+22.804982785" watchObservedRunningTime="2026-07-27 21:34:14.396602924 +0000 UTC m=+22.805024952" Jul 27 21:34:15.186130 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.186106 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:15.186256 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.186151 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:15.186256 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:15.186243 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:15.186379 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:15.186338 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:15.289505 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.289486 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 21:34:15.305093 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.305068 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"0de243660ac89134a2cb0efc226be7734cf99ae248ea205f965fea23c60115a0"} Jul 27 21:34:15.305207 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.305140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"586a8a5dc575cb6d9dd3e6610cb54595421d582c129d7fbb4566de101e28c93c"} Jul 27 21:34:15.305207 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.305199 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"7d61ac838c70c75c7fa80d0917661706dfabbd819b79d18d46101d77340e617f"} Jul 27 21:34:15.306831 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.306804 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" event={"ID":"eeca7b48-3820-4595-88ee-37d4c98d392a","Type":"ContainerStarted","Data":"21b94ad793b93007744cebda9e67115b72fff7a3662881aaaefcc0266a7005e1"} Jul 27 21:34:15.308174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:15.308116 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9mbf" event={"ID":"ea75c0a0-edbc-4a2d-8993-f4377d799d19","Type":"ContainerStarted","Data":"d18936505ecaa5ea59edcd6907d5aff0313b942aeb5401c5271a7c5a0679da9a"} Jul 27 21:34:16.042400 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:16.042274 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:34:15.289499967Z","UUID":"8211bd78-3367-4a3b-ab26-e2ac620f9587","Handler":null,"Name":"","Endpoint":""} Jul 27 21:34:16.044329 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:16.044304 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 21:34:16.044509 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:16.044348 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 21:34:16.187068 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:16.187036 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:16.187238 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:16.187159 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:16.513136 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:16.513104 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:34:16.514139 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:16.513942 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:34:16.529521 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:16.529462 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k9mbf" podStartSLOduration=5.719226025 podStartE2EDuration="24.529441135s" totalImagesPullingTime="18.81021511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:15.325654672 +0000 UTC m=+23.734076704" watchObservedRunningTime="2026-07-27 21:34:16.529441135 +0000 UTC m=+24.937863165" Jul 27 21:34:17.186904 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:17.186825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:17.187054 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:17.186825 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:17.187054 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:17.186935 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:17.187054 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:17.187008 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:17.314578 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:17.314539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" event={"ID":"eeca7b48-3820-4595-88ee-37d4c98d392a","Type":"ContainerStarted","Data":"09a03a432659468a26c8f9d3387ef3256a0533e63c72b2c8ff30a3f4f02f2c29"} Jul 27 21:34:17.318253 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:17.318217 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"739fd948aae902dad8f99a78221d88b0b00a4641f3d52f80511cb93156e6db69"} Jul 27 21:34:17.318434 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:17.318312 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:34:17.318951 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:17.318927 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fmvdv" Jul 27 21:34:17.339779 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:17.339733 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zsccb" podStartSLOduration=4.232749003 podStartE2EDuration="25.339717144s" totalImagesPullingTime="21.106968141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:17.339355995 +0000 UTC m=+25.747778024" watchObservedRunningTime="2026-07-27 21:34:17.339717144 +0000 UTC m=+25.748139172" Jul 27 21:34:18.186927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:18.186880 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:18.187408 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:18.187025 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:19.186499 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:19.186322 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:19.186659 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:19.186361 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:19.186659 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:19.186610 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:19.186795 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:19.186716 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:19.324661 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:19.324575 2556 generic.go:356] "Generic (PLEG): container finished" podID="6d6b8115-32c2-440c-8b91-099f8716b64c" containerID="1f9e37c52e5dc5ee1798f5cfec6ee3c261e8aa43cccd5851602a719adfb06b4b" exitCode=0 Jul 27 21:34:19.325574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:19.324658 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerDied","Data":"1f9e37c52e5dc5ee1798f5cfec6ee3c261e8aa43cccd5851602a719adfb06b4b"} Jul 27 21:34:19.328307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:19.328269 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" event={"ID":"ae79c26d-4dbc-4d9a-aa29-9a07273245c0","Type":"ContainerStarted","Data":"720cff82f9cf513e5582fd68c1bed8cbfb6685b1f21f283aea61eb48adf0623a"} Jul 27 21:34:19.376161 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:19.376106 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" podStartSLOduration=8.495386691 podStartE2EDuration="27.376090323s" totalImagesPullingTime="18.880703632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:19.374883261 +0000 UTC m=+27.783305305" watchObservedRunningTime="2026-07-27 21:34:19.376090323 +0000 UTC m=+27.784512352" Jul 27 21:34:20.186830 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:20.186755 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:20.187021 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:20.186896 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:20.331748 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:20.331713 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:34:20.332124 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:20.331814 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:34:20.332124 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:20.331864 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:34:20.351476 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:20.351119 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:34:20.351919 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:20.351859 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:34:21.077619 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.077424 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-85djc"] Jul 27 21:34:21.077805 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.077751 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:21.077884 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:21.077866 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:21.078111 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.078082 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m24q9"] Jul 27 21:34:21.078208 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.078193 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:21.078294 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:21.078273 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:21.080361 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.080082 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9ch7"] Jul 27 21:34:21.080361 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.080189 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:21.080361 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:21.080322 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:21.335380 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.335342 2556 generic.go:356] "Generic (PLEG): container finished" podID="6d6b8115-32c2-440c-8b91-099f8716b64c" containerID="7c8adb89a632e0b34b18247096abafcb686e95b6853a830d43fc1473121fdfea" exitCode=0 Jul 27 21:34:21.335838 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:21.335433 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerDied","Data":"7c8adb89a632e0b34b18247096abafcb686e95b6853a830d43fc1473121fdfea"} Jul 27 21:34:22.187430 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:22.187388 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:22.187634 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:22.187497 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:23.186250 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:23.186173 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:23.186616 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:23.186284 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:23.186616 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:23.186331 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:23.186616 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:23.186412 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:24.186437 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:24.186349 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:24.186827 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:24.186492 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:24.344218 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:24.344186 2556 generic.go:356] "Generic (PLEG): container finished" podID="6d6b8115-32c2-440c-8b91-099f8716b64c" containerID="d20ba43e6a1ddbe8b7b97d7530e9dabaa824b30e496abe0d67efa55707ef58d3" exitCode=0 Jul 27 21:34:24.344348 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:24.344242 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerDied","Data":"d20ba43e6a1ddbe8b7b97d7530e9dabaa824b30e496abe0d67efa55707ef58d3"} Jul 27 21:34:24.666862 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:24.666822 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:24.751005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:24.750963 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:25.186945 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:25.186916 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:25.187395 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.187043 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:25.187395 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:25.187087 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:25.187395 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.187180 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:25.730604 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:25.730375 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:25.730793 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.730534 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 21:34:25.730793 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.730699 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret podName:8093b04e-e222-499b-b2e2-8bc7b0fc351d nodeName:}" failed. No retries permitted until 2026-07-27 21:34:57.730664486 +0000 UTC m=+66.139086498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret") pod "global-pull-secret-syncer-m24q9" (UID: "8093b04e-e222-499b-b2e2-8bc7b0fc351d") : object "kube-system"/"original-pull-secret" not registered Jul 27 21:34:25.831514 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:25.831475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:25.831514 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:25.831532 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:25.831768 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.831655 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:34:25.831768 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.831746 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs podName:ac1951dc-f8fe-4888-81ff-71d3dd80d587 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:57.83172942 +0000 UTC m=+66.240151428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs") pod "network-metrics-daemon-x9ch7" (UID: "ac1951dc-f8fe-4888-81ff-71d3dd80d587") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 21:34:25.831864 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.831660 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 21:34:25.831864 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.831790 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 21:34:25.831864 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.831804 2556 projected.go:196] Error preparing data for projected volume kube-api-access-t9cvj for pod openshift-network-diagnostics/network-check-target-85djc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:34:25.831864 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:25.831851 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj podName:af0b01fc-decc-4da8-9310-445e5e4e3b89 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:57.831836481 +0000 UTC m=+66.240258493 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9cvj" (UniqueName: "kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj") pod "network-check-target-85djc" (UID: "af0b01fc-decc-4da8-9310-445e5e4e3b89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 21:34:26.186417 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:26.186322 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:26.186569 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:26.186459 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-85djc" podUID="af0b01fc-decc-4da8-9310-445e5e4e3b89" Jul 27 21:34:27.186782 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.186748 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:27.187264 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.186749 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:27.187264 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:27.186892 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9ch7" podUID="ac1951dc-f8fe-4888-81ff-71d3dd80d587" Jul 27 21:34:27.187264 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:27.186966 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m24q9" podUID="8093b04e-e222-499b-b2e2-8bc7b0fc351d" Jul 27 21:34:27.462157 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.462124 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-37.ec2.internal" event="NodeReady" Jul 27 21:34:27.462313 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.462259 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 21:34:27.498980 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.498947 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-777dc78d4c-rndfg"] Jul 27 21:34:27.501412 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.501389 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.504191 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.504164 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 21:34:27.504383 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.504359 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 21:34:27.504506 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.504494 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 21:34:27.505404 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.505387 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6vht4\"" Jul 27 21:34:27.514066 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.512631 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tfdcz"] Jul 27 21:34:27.514066 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.513949 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 21:34:27.515893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.515874 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.519630 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.519614 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 21:34:27.519747 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.519653 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 21:34:27.519747 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.519711 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kb678\"" Jul 27 21:34:27.523151 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.523129 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cg7rm"] Jul 27 21:34:27.525497 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.525475 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777dc78d4c-rndfg"] Jul 27 21:34:27.525619 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.525535 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tfdcz"] Jul 27 21:34:27.525619 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.525605 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:27.530468 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.530440 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 21:34:27.530655 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.530635 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 21:34:27.530793 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.530656 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 21:34:27.530793 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.530706 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7fmv4\"" Jul 27 21:34:27.537368 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.537348 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cg7rm"] Jul 27 21:34:27.646537 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646503 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g97t\" (UniqueName: \"kubernetes.io/projected/8f664142-e58d-4573-8ab9-40738e1a5c70-kube-api-access-7g97t\") pod \"ingress-canary-cg7rm\" (UID: \"8f664142-e58d-4573-8ab9-40738e1a5c70\") " pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:27.646752 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-tmp-dir\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.646752 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646576 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f664142-e58d-4573-8ab9-40738e1a5c70-cert\") pod \"ingress-canary-cg7rm\" (UID: \"8f664142-e58d-4573-8ab9-40738e1a5c70\") " pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:27.646752 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646650 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-installation-pull-secrets\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.646893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646754 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-image-registry-private-configuration\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.646893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646796 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcpt\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-kube-api-access-cjcpt\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.646893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646818 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.646893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646842 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-bound-sa-token\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.646893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646865 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69e19352-5ae6-454e-991e-1565c6672d0e-ca-trust-extracted\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.646893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646879 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-registry-certificates\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.646893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646893 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snfjf\" (UniqueName: \"kubernetes.io/projected/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-kube-api-access-snfjf\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.647152 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646927 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.647152 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.646993 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-config-volume\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.647152 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.647015 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-trusted-ca\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.653623 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.653585 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:27.747805 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69e19352-5ae6-454e-991e-1565c6672d0e-ca-trust-extracted\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.747805 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747765 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-registry-certificates\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.747805 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747791 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snfjf\" (UniqueName: \"kubernetes.io/projected/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-kube-api-access-snfjf\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747876 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-config-volume\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747897 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-trusted-ca\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747916 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g97t\" (UniqueName: \"kubernetes.io/projected/8f664142-e58d-4573-8ab9-40738e1a5c70-kube-api-access-7g97t\") pod \"ingress-canary-cg7rm\" (UID: \"8f664142-e58d-4573-8ab9-40738e1a5c70\") " pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747941 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-tmp-dir\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.747968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f664142-e58d-4573-8ab9-40738e1a5c70-cert\") pod \"ingress-canary-cg7rm\" (UID: \"8f664142-e58d-4573-8ab9-40738e1a5c70\") " pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748029 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-installation-pull-secrets\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748069 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-image-registry-private-configuration\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:27.748081 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748096 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcpt\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-kube-api-access-cjcpt\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748123 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:27.748171 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls podName:e4d5c4d8-db45-4169-9cfc-6368ce4c34a3 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:28.248144498 +0000 UTC m=+36.656566507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls") pod "dns-default-tfdcz" (UID: "e4d5c4d8-db45-4169-9cfc-6368ce4c34a3") : secret "dns-default-metrics-tls" not found Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748198 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-bound-sa-token\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748216 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69e19352-5ae6-454e-991e-1565c6672d0e-ca-trust-extracted\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:27.748258 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:27.748274 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777dc78d4c-rndfg: secret "image-registry-tls" not found Jul 27 21:34:27.748427 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:27.748330 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls podName:69e19352-5ae6-454e-991e-1565c6672d0e nodeName:}" failed. No retries permitted until 2026-07-27 21:34:28.248313788 +0000 UTC m=+36.656735814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls") pod "image-registry-777dc78d4c-rndfg" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e") : secret "image-registry-tls" not found Jul 27 21:34:27.748866 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748457 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-registry-certificates\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748866 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748539 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-config-volume\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.748866 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-trusted-ca\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.748866 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.748799 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-tmp-dir\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.753550 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.753526 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f664142-e58d-4573-8ab9-40738e1a5c70-cert\") pod \"ingress-canary-cg7rm\" (UID: \"8f664142-e58d-4573-8ab9-40738e1a5c70\") " pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:27.753708 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.753534 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-installation-pull-secrets\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.755411 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.755369 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-image-registry-private-configuration\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.757162 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.757116 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g97t\" (UniqueName: \"kubernetes.io/projected/8f664142-e58d-4573-8ab9-40738e1a5c70-kube-api-access-7g97t\") pod \"ingress-canary-cg7rm\" (UID: \"8f664142-e58d-4573-8ab9-40738e1a5c70\") " pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:27.757491 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.757460 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snfjf\" (UniqueName: \"kubernetes.io/projected/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-kube-api-access-snfjf\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:27.757776 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.757753 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-bound-sa-token\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.757879 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.757862 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcpt\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-kube-api-access-cjcpt\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:27.835041 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:27.835005 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cg7rm" Jul 27 21:34:28.186842 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:28.186765 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:28.189599 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:28.189566 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcmmw\"" Jul 27 21:34:28.190005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:28.189797 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 21:34:28.190005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:28.189807 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 21:34:28.253572 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:28.253535 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:28.253745 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:28.253633 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:28.253745 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:28.253708 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:34:28.253745 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:28.253743 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:34:28.253880 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:28.253755 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777dc78d4c-rndfg: secret "image-registry-tls" not found Jul 27 21:34:28.253880 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:28.253776 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls podName:e4d5c4d8-db45-4169-9cfc-6368ce4c34a3 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:29.253757953 +0000 UTC m=+37.662179961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls") pod "dns-default-tfdcz" (UID: "e4d5c4d8-db45-4169-9cfc-6368ce4c34a3") : secret "dns-default-metrics-tls" not found Jul 27 21:34:28.253880 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:28.253822 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls podName:69e19352-5ae6-454e-991e-1565c6672d0e nodeName:}" failed. No retries permitted until 2026-07-27 21:34:29.253811114 +0000 UTC m=+37.662233121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls") pod "image-registry-777dc78d4c-rndfg" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e") : secret "image-registry-tls" not found Jul 27 21:34:28.395803 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:28.395548 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cg7rm"] Jul 27 21:34:28.398928 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:34:28.398902 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f664142_e58d_4573_8ab9_40738e1a5c70.slice/crio-97835a4cdf99e5a0364abe31f3c99fad34ced64e6cd866a8f76e4af9f1b6ac7f WatchSource:0}: Error finding container 97835a4cdf99e5a0364abe31f3c99fad34ced64e6cd866a8f76e4af9f1b6ac7f: Status 404 returned error can't find the container with id 97835a4cdf99e5a0364abe31f3c99fad34ced64e6cd866a8f76e4af9f1b6ac7f Jul 27 21:34:29.009020 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.008932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:29.069626 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.069592 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:29.186690 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.186640 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:29.186877 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.186656 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:29.189651 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.189618 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:34:29.189651 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.189620 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:34:29.189826 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.189654 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9xnkh\"" Jul 27 21:34:29.262611 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.262523 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:29.262611 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.262572 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:29.262812 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:29.262689 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:34:29.262812 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:29.262696 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:34:29.262812 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:29.262720 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777dc78d4c-rndfg: secret "image-registry-tls" not found Jul 27 21:34:29.262812 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:29.262742 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls podName:e4d5c4d8-db45-4169-9cfc-6368ce4c34a3 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:31.262729122 +0000 UTC m=+39.671151131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls") pod "dns-default-tfdcz" (UID: "e4d5c4d8-db45-4169-9cfc-6368ce4c34a3") : secret "dns-default-metrics-tls" not found Jul 27 21:34:29.262812 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:29.262765 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls podName:69e19352-5ae6-454e-991e-1565c6672d0e nodeName:}" failed. No retries permitted until 2026-07-27 21:34:31.262750588 +0000 UTC m=+39.671172596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls") pod "image-registry-777dc78d4c-rndfg" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e") : secret "image-registry-tls" not found Jul 27 21:34:29.360190 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.360162 2556 generic.go:356] "Generic (PLEG): container finished" podID="6d6b8115-32c2-440c-8b91-099f8716b64c" containerID="a101c11b17c258dd9831646b78aa672abe89b53fdc14089d07c83549dfe88c39" exitCode=0 Jul 27 21:34:29.360340 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.360240 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerDied","Data":"a101c11b17c258dd9831646b78aa672abe89b53fdc14089d07c83549dfe88c39"} Jul 27 21:34:29.361402 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.361237 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cg7rm" event={"ID":"8f664142-e58d-4573-8ab9-40738e1a5c70","Type":"ContainerStarted","Data":"97835a4cdf99e5a0364abe31f3c99fad34ced64e6cd866a8f76e4af9f1b6ac7f"} Jul 27 21:34:29.598198 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.598118 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:29.654964 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:29.654936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:30.367005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:30.366960 2556 generic.go:356] "Generic (PLEG): container finished" podID="6d6b8115-32c2-440c-8b91-099f8716b64c" containerID="3a6c822520cdc4e5019d9896fbd775575191e455065ef778f4994b86699932bb" exitCode=0 Jul 27 21:34:30.367623 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:30.367032 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerDied","Data":"3a6c822520cdc4e5019d9896fbd775575191e455065ef778f4994b86699932bb"} Jul 27 21:34:31.158260 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.158221 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:31.254623 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.254587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:31.278409 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.278377 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:31.278577 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.278423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:31.278577 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:31.278539 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:34:31.278577 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:31.278542 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:34:31.278577 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:31.278565 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777dc78d4c-rndfg: secret "image-registry-tls" not found Jul 27 21:34:31.278766 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:31.278598 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls podName:e4d5c4d8-db45-4169-9cfc-6368ce4c34a3 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:35.278577762 +0000 UTC m=+43.686999770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls") pod "dns-default-tfdcz" (UID: "e4d5c4d8-db45-4169-9cfc-6368ce4c34a3") : secret "dns-default-metrics-tls" not found Jul 27 21:34:31.278766 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:31.278621 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls podName:69e19352-5ae6-454e-991e-1565c6672d0e nodeName:}" failed. No retries permitted until 2026-07-27 21:34:35.278608194 +0000 UTC m=+43.687030218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls") pod "image-registry-777dc78d4c-rndfg" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e") : secret "image-registry-tls" not found Jul 27 21:34:31.355243 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.355207 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:31.373890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.373859 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" event={"ID":"6d6b8115-32c2-440c-8b91-099f8716b64c","Type":"ContainerStarted","Data":"f7fed93e014318ee532740e10099c768f2bda52256ca9fe2e2a3d1516864a110"} Jul 27 21:34:31.401519 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.401463 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bwp4t" podStartSLOduration=8.487971431 podStartE2EDuration="39.401443916s" totalImagesPullingTime="30.913472485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:31.400551107 +0000 UTC m=+39.808973139" watchObservedRunningTime="2026-07-27 21:34:31.401443916 +0000 UTC m=+39.809865947" Jul 27 21:34:31.987229 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:31.987200 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:32.073371 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:32.073336 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:32.188296 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:32.188263 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:32.284281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:32.284183 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:32.377893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:32.377861 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cg7rm" event={"ID":"8f664142-e58d-4573-8ab9-40738e1a5c70","Type":"ContainerStarted","Data":"e90414f5536bb0c374538709b371be2746f727683ead4cca5baaf3d2429d0515"} Jul 27 21:34:33.187807 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:33.187770 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:33.261905 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:33.261863 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:34.134713 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:34.134655 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:34.706640 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:34.706610 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:34.765003 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:34.764967 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:35.310372 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.310335 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:35.310857 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.310383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:35.310857 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.310482 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:34:35.310857 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.310517 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:34:35.310857 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.310536 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls podName:e4d5c4d8-db45-4169-9cfc-6368ce4c34a3 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:43.310522054 +0000 UTC m=+51.718944062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls") pod "dns-default-tfdcz" (UID: "e4d5c4d8-db45-4169-9cfc-6368ce4c34a3") : secret "dns-default-metrics-tls" not found Jul 27 21:34:35.310857 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.310542 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777dc78d4c-rndfg: secret "image-registry-tls" not found Jul 27 21:34:35.310857 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.310602 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls podName:69e19352-5ae6-454e-991e-1565c6672d0e nodeName:}" failed. No retries permitted until 2026-07-27 21:34:43.310582905 +0000 UTC m=+51.719004923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls") pod "image-registry-777dc78d4c-rndfg" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e") : secret "image-registry-tls" not found Jul 27 21:34:35.690611 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.690488 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cg7rm" podStartSLOduration=5.029055275 podStartE2EDuration="8.690466898s" totalImagesPullingTime="3.661411623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:32.410933125 +0000 UTC m=+40.819355154" watchObservedRunningTime="2026-07-27 21:34:35.690466898 +0000 UTC m=+44.098888928" Jul 27 21:34:35.690939 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.690916 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zswrh"] Jul 27 21:34:35.720406 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.720371 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zswrh"] Jul 27 21:34:35.720549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.720496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.723281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.723256 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 21:34:35.723423 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.723257 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 21:34:35.723423 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.723262 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 21:34:35.723423 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.723318 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d6xkx\"" Jul 27 21:34:35.724322 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.724306 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 21:34:35.731503 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.731477 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:35.789570 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.789534 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:35.812899 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.812870 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36d73831-2e20-4aab-8964-c5850d2dc0d0-crio-socket\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.812975 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.812903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36d73831-2e20-4aab-8964-c5850d2dc0d0-data-volume\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.812975 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.812924 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.813051 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.813013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.813090 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.813063 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssng\" (UniqueName: \"kubernetes.io/projected/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-api-access-lssng\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.914389 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.914349 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36d73831-2e20-4aab-8964-c5850d2dc0d0-crio-socket\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.914389 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.914388 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36d73831-2e20-4aab-8964-c5850d2dc0d0-data-volume\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.914595 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.914404 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.914595 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.914445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.914595 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.914484 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssng\" (UniqueName: \"kubernetes.io/projected/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-api-access-lssng\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.914595 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.914544 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:34:35.914595 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.914570 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:34:35.914827 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.914600 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36d73831-2e20-4aab-8964-c5850d2dc0d0-crio-socket\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.914827 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.914637 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:36.414614184 +0000 UTC m=+44.823036196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : secret "insights-runtime-extractor-tls" not found Jul 27 21:34:35.914827 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:35.914657 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:36.414647263 +0000 UTC m=+44.823069270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : configmap "kube-rbac-proxy" not found Jul 27 21:34:35.914827 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.914786 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36d73831-2e20-4aab-8964-c5850d2dc0d0-data-volume\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:35.924500 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:35.924468 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssng\" (UniqueName: \"kubernetes.io/projected/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-api-access-lssng\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:36.172242 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.172205 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:36.272899 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.272865 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:36.387803 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.387769 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:36.416502 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.416469 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs"] Jul 27 21:34:36.420697 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.420654 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:36.420867 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.420725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:36.420867 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:36.420817 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:34:36.420867 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:36.420827 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:34:36.420867 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:36.420867 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:37.420852634 +0000 UTC m=+45.829274642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : configmap "kube-rbac-proxy" not found Jul 27 21:34:36.421031 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:36.420885 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:37.420876692 +0000 UTC m=+45.829298701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : secret "insights-runtime-extractor-tls" not found Jul 27 21:34:36.434705 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.434631 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" Jul 27 21:34:36.438167 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.438148 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 27 21:34:36.438313 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.438296 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r692t\"" Jul 27 21:34:36.441062 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.440944 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 27 21:34:36.443927 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.443904 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs"] Jul 27 21:34:36.518329 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.518299 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:36.618130 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.618081 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:36.622102 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.622071 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxs8\" (UniqueName: \"kubernetes.io/projected/8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec-kube-api-access-4zxs8\") pod \"migrator-6865c9967-mqxcs\" (UID: \"8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" Jul 27 21:34:36.722913 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.722882 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxs8\" (UniqueName: \"kubernetes.io/projected/8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec-kube-api-access-4zxs8\") pod \"migrator-6865c9967-mqxcs\" (UID: \"8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" Jul 27 21:34:36.731656 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.731619 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxs8\" (UniqueName: \"kubernetes.io/projected/8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec-kube-api-access-4zxs8\") pod \"migrator-6865c9967-mqxcs\" (UID: \"8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" Jul 27 21:34:36.743507 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.743479 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" Jul 27 21:34:36.896786 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:36.896750 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs"] Jul 27 21:34:36.899824 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:34:36.899791 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab1a4c3_f32a_4f5a_8a76_319e225ee3ec.slice/crio-60bbe7d15bbaa216b30f20b5ef5d8f0bf56866280c77b0d309d5c0d2415b27cb WatchSource:0}: Error finding container 60bbe7d15bbaa216b30f20b5ef5d8f0bf56866280c77b0d309d5c0d2415b27cb: Status 404 returned error can't find the container with id 60bbe7d15bbaa216b30f20b5ef5d8f0bf56866280c77b0d309d5c0d2415b27cb Jul 27 21:34:37.391350 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:37.391317 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" event={"ID":"8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec","Type":"ContainerStarted","Data":"60bbe7d15bbaa216b30f20b5ef5d8f0bf56866280c77b0d309d5c0d2415b27cb"} Jul 27 21:34:37.427895 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:37.427860 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:37.428018 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:37.427919 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:37.428076 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:37.428023 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:34:37.428076 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:37.428026 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:34:37.428148 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:37.428082 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:39.42806567 +0000 UTC m=+47.836487678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : configmap "kube-rbac-proxy" not found Jul 27 21:34:37.428148 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:37.428096 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:39.428090823 +0000 UTC m=+47.836512832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : secret "insights-runtime-extractor-tls" not found Jul 27 21:34:37.482663 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:37.482638 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qwnv7_9a5f0961-958b-4d25-846a-6320dc6a87be/dns-node-resolver/0.log" Jul 27 21:34:38.043649 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.043619 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-frqnm"] Jul 27 21:34:38.058809 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.058761 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-frqnm"] Jul 27 21:34:38.058974 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.058891 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.061585 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.061563 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 21:34:38.062688 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.062645 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 21:34:38.062688 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.062658 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 21:34:38.062688 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.062645 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jt2nq\"" Jul 27 21:34:38.062893 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.062704 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 21:34:38.078643 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.078598 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:38.142642 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.142592 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:38.234265 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.234228 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68662356-f347-44a7-a3e3-4b02c85a65b5-signing-key\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.234439 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.234296 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68662356-f347-44a7-a3e3-4b02c85a65b5-signing-cabundle\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.234439 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.234355 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgms\" (UniqueName: \"kubernetes.io/projected/68662356-f347-44a7-a3e3-4b02c85a65b5-kube-api-access-ndgms\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.282168 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.282139 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7rkt6_edb9df30-5601-4bcc-9d00-3d531da72dd5/node-ca/0.log" Jul 27 21:34:38.335792 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.335712 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgms\" (UniqueName: \"kubernetes.io/projected/68662356-f347-44a7-a3e3-4b02c85a65b5-kube-api-access-ndgms\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.335940 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.335840 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68662356-f347-44a7-a3e3-4b02c85a65b5-signing-key\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.335940 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.335891 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68662356-f347-44a7-a3e3-4b02c85a65b5-signing-cabundle\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.336595 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.336569 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/68662356-f347-44a7-a3e3-4b02c85a65b5-signing-cabundle\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.338460 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.338438 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/68662356-f347-44a7-a3e3-4b02c85a65b5-signing-key\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.344945 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.344924 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgms\" (UniqueName: \"kubernetes.io/projected/68662356-f347-44a7-a3e3-4b02c85a65b5-kube-api-access-ndgms\") pod \"service-ca-658d7b7c78-frqnm\" (UID: \"68662356-f347-44a7-a3e3-4b02c85a65b5\") " pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.368696 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.368259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" Jul 27 21:34:38.512776 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.512217 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-frqnm"] Jul 27 21:34:38.516891 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:34:38.516858 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68662356_f347_44a7_a3e3_4b02c85a65b5.slice/crio-47dce4638f545e6d83a6af5e466d49ae020707473069c45b927cfbd68a58fa4f WatchSource:0}: Error finding container 47dce4638f545e6d83a6af5e466d49ae020707473069c45b927cfbd68a58fa4f: Status 404 returned error can't find the container with id 47dce4638f545e6d83a6af5e466d49ae020707473069c45b927cfbd68a58fa4f Jul 27 21:34:38.997155 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:38.997054 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:39.081994 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:39.081937 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cg7rm_8f664142-e58d-4573-8ab9-40738e1a5c70/serve-healthcheck-canary/0.log" Jul 27 21:34:39.398174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:39.398073 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" event={"ID":"68662356-f347-44a7-a3e3-4b02c85a65b5","Type":"ContainerStarted","Data":"47dce4638f545e6d83a6af5e466d49ae020707473069c45b927cfbd68a58fa4f"} Jul 27 21:34:39.400323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:39.400291 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" event={"ID":"8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec","Type":"ContainerStarted","Data":"6212eec5ba1ceb7fbfb440d6e35c0bf4ab250067907ee11e04d92c10fc31db43"} Jul 27 21:34:39.400441 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:39.400349 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" event={"ID":"8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec","Type":"ContainerStarted","Data":"e290c2f6bcdf9514f9c75f08fe786288911d76fc86acbc75bb92d26e041a20b1"} Jul 27 21:34:39.420811 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:39.420755 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mqxcs" podStartSLOduration=1.5909580559999998 podStartE2EDuration="3.420734677s" totalImagesPullingTime="1.829776621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:39.419748101 +0000 UTC m=+47.828170131" watchObservedRunningTime="2026-07-27 21:34:39.420734677 +0000 UTC m=+47.829156709" Jul 27 21:34:39.446301 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:39.446260 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:39.446469 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:39.446350 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:39.446469 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:39.446418 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:34:39.446562 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:39.446482 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:43.446466668 +0000 UTC m=+51.854888679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : secret "insights-runtime-extractor-tls" not found Jul 27 21:34:39.446562 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:39.446479 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 21:34:39.446562 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:39.446546 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:43.44652895 +0000 UTC m=+51.854950979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : configmap "kube-rbac-proxy" not found Jul 27 21:34:41.407597 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:41.407560 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" event={"ID":"68662356-f347-44a7-a3e3-4b02c85a65b5","Type":"ContainerStarted","Data":"54f19ad86ef37e15c46ddfd9386014fcbeb4fd6548600c2cd9d553250a4cc3f2"} Jul 27 21:34:41.430316 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:41.430263 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-frqnm" podStartSLOduration=1.162247079 podStartE2EDuration="3.43024746s" totalImagesPullingTime="2.268000381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:41.429413627 +0000 UTC m=+49.837835658" watchObservedRunningTime="2026-07-27 21:34:41.43024746 +0000 UTC m=+49.838669489" Jul 27 21:34:41.434317 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:41.434286 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:42.216848 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:42.216804 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:42.332915 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:42.332870 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:42.635415 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:42.635316 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:42.735281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:42.735249 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:42.834975 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:42.834939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:42.934367 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:42.934273 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:43.034775 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:43.034730 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:43.377829 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:43.377792 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:43.378010 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:43.377846 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:43.378010 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:43.377951 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 21:34:43.378010 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:43.377955 2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 21:34:43.378010 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:43.377974 2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-777dc78d4c-rndfg: secret "image-registry-tls" not found Jul 27 21:34:43.378148 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:43.378019 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls podName:69e19352-5ae6-454e-991e-1565c6672d0e nodeName:}" failed. No retries permitted until 2026-07-27 21:34:59.377998055 +0000 UTC m=+67.786420064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls") pod "image-registry-777dc78d4c-rndfg" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e") : secret "image-registry-tls" not found Jul 27 21:34:43.378148 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:43.378035 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls podName:e4d5c4d8-db45-4169-9cfc-6368ce4c34a3 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:59.378027406 +0000 UTC m=+67.786449413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls") pod "dns-default-tfdcz" (UID: "e4d5c4d8-db45-4169-9cfc-6368ce4c34a3") : secret "dns-default-metrics-tls" not found Jul 27 21:34:43.478382 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:43.478335 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:43.478565 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:43.478417 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:43.478565 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:43.478487 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 21:34:43.478565 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:34:43.478557 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls podName:36d73831-2e20-4aab-8964-c5850d2dc0d0 nodeName:}" failed. No retries permitted until 2026-07-27 21:34:51.478540758 +0000 UTC m=+59.886962766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zswrh" (UID: "36d73831-2e20-4aab-8964-c5850d2dc0d0") : secret "insights-runtime-extractor-tls" not found Jul 27 21:34:43.479034 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:43.479016 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36d73831-2e20-4aab-8964-c5850d2dc0d0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:43.912320 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:43.912287 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:44.096239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:44.096190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:44.492895 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:44.492855 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:44.696278 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:44.696230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:45.299112 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:45.299077 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:45.695160 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:45.695131 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:45.892763 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:45.892729 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:46.122566 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:46.122449 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:46.217006 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:46.216970 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:46.317135 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:46.317087 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:46.697189 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:46.697159 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:46.755047 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:46.755010 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:47.297112 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:47.297071 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:47.699276 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:47.699240 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:47.800941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:47.800908 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:47.900784 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:47.900747 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:48.106301 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:48.106204 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:48.160792 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:48.160752 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:48.294620 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:48.294586 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:48.696264 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:48.696230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:49.016262 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:49.016162 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:49.116861 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:49.116828 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:49.503082 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:49.503045 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:49.698943 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:49.698903 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:49.758400 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:49.758295 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:50.107356 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:50.107271 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:50.166362 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:50.166328 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:50.300929 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:50.300895 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:50.699759 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:50.699724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:51.302920 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:51.302885 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:51.544371 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:51.544332 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:51.546918 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:51.546884 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36d73831-2e20-4aab-8964-c5850d2dc0d0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zswrh\" (UID: \"36d73831-2e20-4aab-8964-c5850d2dc0d0\") " pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:51.629641 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:51.629554 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zswrh" Jul 27 21:34:51.771043 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:51.771010 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zswrh"] Jul 27 21:34:51.774245 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:34:51.774219 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d73831_2e20_4aab_8964_c5850d2dc0d0.slice/crio-595dae84b0da86837f00815aa34bb5caba39639c30e576dc5399c3ec503f242e WatchSource:0}: Error finding container 595dae84b0da86837f00815aa34bb5caba39639c30e576dc5399c3ec503f242e: Status 404 returned error can't find the container with id 595dae84b0da86837f00815aa34bb5caba39639c30e576dc5399c3ec503f242e Jul 27 21:34:51.906037 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:51.905945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:52.212546 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:52.212459 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:52.313751 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:52.313695 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:52.353182 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:52.353149 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cks79" Jul 27 21:34:52.437888 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:52.437809 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zswrh" event={"ID":"36d73831-2e20-4aab-8964-c5850d2dc0d0","Type":"ContainerStarted","Data":"595dae84b0da86837f00815aa34bb5caba39639c30e576dc5399c3ec503f242e"} Jul 27 21:34:52.504448 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:52.504329 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:52.911152 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:52.911068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:57.454923 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.454875 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zswrh" event={"ID":"36d73831-2e20-4aab-8964-c5850d2dc0d0","Type":"ContainerStarted","Data":"7d6297ca5033754fb1b27c4b7ba16d069f75e3009b01a92f743288d7f0f2fad7"} Jul 27 21:34:57.801043 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.800943 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:57.803574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.803551 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 21:34:57.814314 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.814283 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8093b04e-e222-499b-b2e2-8bc7b0fc351d-original-pull-secret\") pod \"global-pull-secret-syncer-m24q9\" (UID: \"8093b04e-e222-499b-b2e2-8bc7b0fc351d\") " pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:57.901849 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.901803 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:57.902022 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.901866 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:57.905755 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.905713 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 21:34:57.906836 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.906810 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 21:34:57.915392 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.915360 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac1951dc-f8fe-4888-81ff-71d3dd80d587-metrics-certs\") pod \"network-metrics-daemon-x9ch7\" (UID: \"ac1951dc-f8fe-4888-81ff-71d3dd80d587\") " pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:57.916223 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.916203 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 21:34:57.926033 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.926009 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9cvj\" (UniqueName: \"kubernetes.io/projected/af0b01fc-decc-4da8-9310-445e5e4e3b89-kube-api-access-t9cvj\") pod \"network-check-target-85djc\" (UID: \"af0b01fc-decc-4da8-9310-445e5e4e3b89\") " pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:57.996163 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:57.996128 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m24q9" Jul 27 21:34:58.004864 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.004839 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9xnkh\"" Jul 27 21:34:58.011803 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.011780 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9ch7" Jul 27 21:34:58.067650 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.067559 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:58.168431 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.168399 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:58.200238 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.200195 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kcmmw\"" Jul 27 21:34:58.207638 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.207611 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:34:58.285093 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.285054 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:58.462750 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.462716 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:58.523740 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.523705 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:58.860770 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.860661 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:58.922605 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:58.922568 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:59.057010 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.056968 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:59.166550 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.166458 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:59.414051 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.414015 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:59.414051 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.414067 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:59.417141 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.417072 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4d5c4d8-db45-4169-9cfc-6368ce4c34a3-metrics-tls\") pod \"dns-default-tfdcz\" (UID: \"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3\") " pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:59.417256 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.417132 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"image-registry-777dc78d4c-rndfg\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:59.619816 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.619780 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6vht4\"" Jul 27 21:34:59.626855 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.626830 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:34:59.629167 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.629142 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kb678\"" Jul 27 21:34:59.637205 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.637140 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tfdcz" Jul 27 21:34:59.672467 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.672370 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:34:59.919643 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:34:59.919606 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:00.019861 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.019824 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:00.234775 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.234727 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m24q9"] Jul 27 21:35:00.253549 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.253517 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9ch7"] Jul 27 21:35:00.256441 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.256416 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tfdcz"] Jul 27 21:35:00.285287 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.285194 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-85djc"] Jul 27 21:35:00.373412 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:00.373365 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8093b04e_e222_499b_b2e2_8bc7b0fc351d.slice/crio-1ed2622e2a6007e0a458850c6caefad61a85432be335c383ba51cc6ab81f4b4d WatchSource:0}: Error finding container 1ed2622e2a6007e0a458850c6caefad61a85432be335c383ba51cc6ab81f4b4d: Status 404 returned error can't find the container with id 1ed2622e2a6007e0a458850c6caefad61a85432be335c383ba51cc6ab81f4b4d Jul 27 21:35:00.374157 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:00.373968 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1951dc_f8fe_4888_81ff_71d3dd80d587.slice/crio-1d6801f5a554c51cf85ddfab35eed2be9cf0c84e195b9031b2407a58f176a470 WatchSource:0}: Error finding container 1d6801f5a554c51cf85ddfab35eed2be9cf0c84e195b9031b2407a58f176a470: Status 404 returned error can't find the container with id 1d6801f5a554c51cf85ddfab35eed2be9cf0c84e195b9031b2407a58f176a470 Jul 27 21:35:00.375186 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:00.375159 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d5c4d8_db45_4169_9cfc_6368ce4c34a3.slice/crio-92ddda54b2cdb0cc6c2aebbc2a48db0f057cbf36af57663b4c568c1935be132a WatchSource:0}: Error finding container 92ddda54b2cdb0cc6c2aebbc2a48db0f057cbf36af57663b4c568c1935be132a: Status 404 returned error can't find the container with id 92ddda54b2cdb0cc6c2aebbc2a48db0f057cbf36af57663b4c568c1935be132a Jul 27 21:35:00.375623 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:00.375588 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0b01fc_decc_4da8_9310_445e5e4e3b89.slice/crio-abe211ccd5300673fe715ee9db6d408cbc659886d3e81b963664e37f2c5cc7ed WatchSource:0}: Error finding container abe211ccd5300673fe715ee9db6d408cbc659886d3e81b963664e37f2c5cc7ed: Status 404 returned error can't find the container with id abe211ccd5300673fe715ee9db6d408cbc659886d3e81b963664e37f2c5cc7ed Jul 27 21:35:00.465155 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.465118 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9ch7" event={"ID":"ac1951dc-f8fe-4888-81ff-71d3dd80d587","Type":"ContainerStarted","Data":"1d6801f5a554c51cf85ddfab35eed2be9cf0c84e195b9031b2407a58f176a470"} Jul 27 21:35:00.466380 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.466353 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfdcz" event={"ID":"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3","Type":"ContainerStarted","Data":"92ddda54b2cdb0cc6c2aebbc2a48db0f057cbf36af57663b4c568c1935be132a"} Jul 27 21:35:00.467782 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.467753 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-85djc" event={"ID":"af0b01fc-decc-4da8-9310-445e5e4e3b89","Type":"ContainerStarted","Data":"abe211ccd5300673fe715ee9db6d408cbc659886d3e81b963664e37f2c5cc7ed"} Jul 27 21:35:00.468982 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.468957 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m24q9" event={"ID":"8093b04e-e222-499b-b2e2-8bc7b0fc351d","Type":"ContainerStarted","Data":"1ed2622e2a6007e0a458850c6caefad61a85432be335c383ba51cc6ab81f4b4d"} Jul 27 21:35:00.493403 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:00.493375 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-777dc78d4c-rndfg"] Jul 27 21:35:00.496330 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:00.496303 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e19352_5ae6_454e_991e_1565c6672d0e.slice/crio-20b7a42156824b11c804e0bca03a8e9a3ae4f5df1eefde91481ecea6f7f2f939 WatchSource:0}: Error finding container 20b7a42156824b11c804e0bca03a8e9a3ae4f5df1eefde91481ecea6f7f2f939: Status 404 returned error can't find the container with id 20b7a42156824b11c804e0bca03a8e9a3ae4f5df1eefde91481ecea6f7f2f939 Jul 27 21:35:01.107969 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:01.105877 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:01.168732 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:01.168691 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:01.488478 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:01.486566 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zswrh" event={"ID":"36d73831-2e20-4aab-8964-c5850d2dc0d0","Type":"ContainerStarted","Data":"22558fc36dc7664d66caed8a35a2a37093d5b945f639b779442c0f3e9ae1d551"} Jul 27 21:35:01.493703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:01.493237 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" event={"ID":"69e19352-5ae6-454e-991e-1565c6672d0e","Type":"ContainerStarted","Data":"0e26665c367f16f7af6a799ca4eaffae9c50cae4b334b4240737204964d12da9"} Jul 27 21:35:01.493703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:01.493292 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" event={"ID":"69e19352-5ae6-454e-991e-1565c6672d0e","Type":"ContainerStarted","Data":"20b7a42156824b11c804e0bca03a8e9a3ae4f5df1eefde91481ecea6f7f2f939"} Jul 27 21:35:01.493703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:01.493448 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:35:01.516330 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:01.515286 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" podStartSLOduration=51.515264231 podStartE2EDuration="51.515264231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:01.514719392 +0000 UTC m=+69.923141424" watchObservedRunningTime="2026-07-27 21:35:01.515264231 +0000 UTC m=+69.923686253" Jul 27 21:35:02.502772 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:02.502740 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9ch7" event={"ID":"ac1951dc-f8fe-4888-81ff-71d3dd80d587","Type":"ContainerStarted","Data":"16ab6790ee3e80541cc3cb4d07748cb827a694dcd5c689b90d407f06b86bb4ee"} Jul 27 21:35:03.174393 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.174348 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk"] Jul 27 21:35:03.177521 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.177495 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.181395 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.181362 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s"] Jul 27 21:35:03.182851 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.182826 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 21:35:03.182973 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.182870 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g6p5f\"" Jul 27 21:35:03.185087 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.185060 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 21:35:03.185503 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.185483 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 21:35:03.185719 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.185308 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 27 21:35:03.187278 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.186736 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.189168 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.189148 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 21:35:03.191895 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.191277 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 21:35:03.191895 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.191473 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 21:35:03.191895 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.191686 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 21:35:03.192937 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.192915 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk"] Jul 27 21:35:03.209199 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.209174 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s"] Jul 27 21:35:03.234772 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.234739 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-777dc78d4c-rndfg"] Jul 27 21:35:03.243367 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243332 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.243541 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243386 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-hub\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.243541 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243433 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.243541 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243466 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.243541 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243500 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/790f9011-4695-4b70-8cf6-dcff8a0e6115-kube-api-access-t65tt\") pod \"managed-serviceaccount-addon-agent-5655b9d984-xd8tk\" (UID: \"790f9011-4695-4b70-8cf6-dcff8a0e6115\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.243541 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243526 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-ca\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.243832 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzd7p\" (UniqueName: \"kubernetes.io/projected/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-kube-api-access-vzd7p\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.243832 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.243588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/790f9011-4695-4b70-8cf6-dcff8a0e6115-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5655b9d984-xd8tk\" (UID: \"790f9011-4695-4b70-8cf6-dcff8a0e6115\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.342612 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.342516 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:03.344105 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344076 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.344251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344118 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/790f9011-4695-4b70-8cf6-dcff8a0e6115-kube-api-access-t65tt\") pod \"managed-serviceaccount-addon-agent-5655b9d984-xd8tk\" (UID: \"790f9011-4695-4b70-8cf6-dcff8a0e6115\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.344251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344142 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-ca\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.344251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344168 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzd7p\" (UniqueName: \"kubernetes.io/projected/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-kube-api-access-vzd7p\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.344251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/790f9011-4695-4b70-8cf6-dcff8a0e6115-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5655b9d984-xd8tk\" (UID: \"790f9011-4695-4b70-8cf6-dcff8a0e6115\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.344251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344246 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.344481 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344271 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-hub\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.344481 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.344318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.345178 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.345105 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.347805 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.347778 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.348232 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.348173 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-ca\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.349496 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.349410 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-hub\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.349623 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.349602 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.349792 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.349745 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/790f9011-4695-4b70-8cf6-dcff8a0e6115-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5655b9d984-xd8tk\" (UID: \"790f9011-4695-4b70-8cf6-dcff8a0e6115\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.371463 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.371428 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65tt\" (UniqueName: \"kubernetes.io/projected/790f9011-4695-4b70-8cf6-dcff8a0e6115-kube-api-access-t65tt\") pod \"managed-serviceaccount-addon-agent-5655b9d984-xd8tk\" (UID: \"790f9011-4695-4b70-8cf6-dcff8a0e6115\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.372227 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.372202 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzd7p\" (UniqueName: \"kubernetes.io/projected/8cd11139-4a15-4a13-b231-aa7f3c1f27fc-kube-api-access-vzd7p\") pod \"cluster-proxy-proxy-agent-6996759c8f-gg76s\" (UID: \"8cd11139-4a15-4a13-b231-aa7f3c1f27fc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.396703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.396649 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-797cb7d859-wmrhq"] Jul 27 21:35:03.400374 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.400342 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.426694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.426174 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797cb7d859-wmrhq"] Jul 27 21:35:03.445348 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445312 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-installation-pull-secrets\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.445515 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445367 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-ca-trust-extracted\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.445515 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445412 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-bound-sa-token\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.445515 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445438 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnjv\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-kube-api-access-sbnjv\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.445515 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445512 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-image-registry-private-configuration\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.445681 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-trusted-ca\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.445681 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445586 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-registry-certificates\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.445681 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.445615 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-registry-tls\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.465758 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.465710 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:03.507100 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.507062 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" Jul 27 21:35:03.514993 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.514967 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" Jul 27 21:35:03.547009 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.546971 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-image-registry-private-configuration\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547181 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547036 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-trusted-ca\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547181 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547110 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-registry-certificates\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547181 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547155 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-registry-tls\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547337 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547216 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-installation-pull-secrets\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547337 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547244 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-ca-trust-extracted\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547337 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-bound-sa-token\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547337 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547302 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnjv\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-kube-api-access-sbnjv\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.547945 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.547917 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-ca-trust-extracted\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.548186 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.548163 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-trusted-ca\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.549040 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.549017 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-registry-certificates\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.550634 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.550593 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-image-registry-private-configuration\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.550752 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.550661 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-installation-pull-secrets\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.551096 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.551074 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-registry-tls\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.563920 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.563879 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-bound-sa-token\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.566657 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.566632 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnjv\" (UniqueName: \"kubernetes.io/projected/007c7d5a-3a89-42e7-a4aa-d24ab75177d4-kube-api-access-sbnjv\") pod \"image-registry-797cb7d859-wmrhq\" (UID: \"007c7d5a-3a89-42e7-a4aa-d24ab75177d4\") " pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.722703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.722582 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:03.860118 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.860082 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:03.962384 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:03.962333 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:04.720786 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:04.720749 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:04.786152 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:04.786103 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:06.548601 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:06.548566 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:06.705896 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:06.705840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:07.069393 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.069360 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:07.295045 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.295011 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:07.400451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.400411 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:07.497854 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.496990 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:07.548021 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.547986 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-85djc" event={"ID":"af0b01fc-decc-4da8-9310-445e5e4e3b89","Type":"ContainerStarted","Data":"3372d8a06f830c6c7972fb4b47fb485b16c86b291d8c71984e7e76701d78bb9f"} Jul 27 21:35:07.548592 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.548568 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:35:07.550656 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.550633 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9ch7" event={"ID":"ac1951dc-f8fe-4888-81ff-71d3dd80d587","Type":"ContainerStarted","Data":"6ebb4d763237248fd392c6d672c8af17bc7370499c1ba3e68f2eb512b8baebb3"} Jul 27 21:35:07.560696 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.559841 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s"] Jul 27 21:35:07.563965 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:07.563628 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cd11139_4a15_4a13_b231_aa7f3c1f27fc.slice/crio-65862e43d7a77b44f25cd268b85234c14353f52b244b61bbed9240fc2c436ade WatchSource:0}: Error finding container 65862e43d7a77b44f25cd268b85234c14353f52b244b61bbed9240fc2c436ade: Status 404 returned error can't find the container with id 65862e43d7a77b44f25cd268b85234c14353f52b244b61bbed9240fc2c436ade Jul 27 21:35:07.585559 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.585496 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk"] Jul 27 21:35:07.586211 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.586133 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-85djc" podStartSLOduration=68.62302186 podStartE2EDuration="1m15.586114873s" totalImagesPullingTime="6.963093013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:07.579413498 +0000 UTC m=+75.987835549" watchObservedRunningTime="2026-07-27 21:35:07.586114873 +0000 UTC m=+75.994536883" Jul 27 21:35:07.587727 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.587655 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797cb7d859-wmrhq"] Jul 27 21:35:07.589251 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:07.588901 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod790f9011_4695_4b70_8cf6_dcff8a0e6115.slice/crio-8668a0410a18a49763b0a3dcc22453b20ec11061c178efcfb2729522fa43cecc WatchSource:0}: Error finding container 8668a0410a18a49763b0a3dcc22453b20ec11061c178efcfb2729522fa43cecc: Status 404 returned error can't find the container with id 8668a0410a18a49763b0a3dcc22453b20ec11061c178efcfb2729522fa43cecc Jul 27 21:35:07.590156 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.589901 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:07.595250 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:07.594876 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007c7d5a_3a89_42e7_a4aa_d24ab75177d4.slice/crio-d953f76f4189e0a13b821f3ab2f03f7d6da80619fb914fbbc5443d0fbc0b0a05 WatchSource:0}: Error finding container d953f76f4189e0a13b821f3ab2f03f7d6da80619fb914fbbc5443d0fbc0b0a05: Status 404 returned error can't find the container with id d953f76f4189e0a13b821f3ab2f03f7d6da80619fb914fbbc5443d0fbc0b0a05 Jul 27 21:35:07.604083 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.603882 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x9ch7" podStartSLOduration=73.911909233 podStartE2EDuration="1m15.60386262s" totalImagesPullingTime="1.691953387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:07.601745047 +0000 UTC m=+76.010167076" watchObservedRunningTime="2026-07-27 21:35:07.60386262 +0000 UTC m=+76.012284650" Jul 27 21:35:07.675346 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:07.675282 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:08.243132 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.243059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:08.559463 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.558532 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zswrh" event={"ID":"36d73831-2e20-4aab-8964-c5850d2dc0d0","Type":"ContainerStarted","Data":"b194a39aa2d5fa9a108bc8337720d5f9bc8e0be04708a47b70d44702ac33f0ba"} Jul 27 21:35:08.561957 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.561927 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfdcz" event={"ID":"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3","Type":"ContainerStarted","Data":"f0c00a4526993ba409f7ffe94daf2fc1f0c087f03e6fd1ff43d975bbf2815c89"} Jul 27 21:35:08.562094 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.561986 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tfdcz" event={"ID":"e4d5c4d8-db45-4169-9cfc-6368ce4c34a3","Type":"ContainerStarted","Data":"af77e88d2fca9d26177c21b7615f2015208fc7166454878dfe0fb4f3e70817ee"} Jul 27 21:35:08.562558 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.562537 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tfdcz" Jul 27 21:35:08.565008 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.564987 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" event={"ID":"790f9011-4695-4b70-8cf6-dcff8a0e6115","Type":"ContainerStarted","Data":"8668a0410a18a49763b0a3dcc22453b20ec11061c178efcfb2729522fa43cecc"} Jul 27 21:35:08.566873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.566849 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" event={"ID":"007c7d5a-3a89-42e7-a4aa-d24ab75177d4","Type":"ContainerStarted","Data":"6564bbf49091dbb7be36133a64313107242c02dd83e09df994b925d0b95270d6"} Jul 27 21:35:08.566961 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.566893 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" event={"ID":"007c7d5a-3a89-42e7-a4aa-d24ab75177d4","Type":"ContainerStarted","Data":"d953f76f4189e0a13b821f3ab2f03f7d6da80619fb914fbbc5443d0fbc0b0a05"} Jul 27 21:35:08.567413 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.567359 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:08.569603 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.569579 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m24q9" event={"ID":"8093b04e-e222-499b-b2e2-8bc7b0fc351d","Type":"ContainerStarted","Data":"a24ae17b1c03467be4fd8a9c407ee25eeeda34dd5f096ca8784609004ff553c6"} Jul 27 21:35:08.571314 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.571292 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" event={"ID":"8cd11139-4a15-4a13-b231-aa7f3c1f27fc","Type":"ContainerStarted","Data":"65862e43d7a77b44f25cd268b85234c14353f52b244b61bbed9240fc2c436ade"} Jul 27 21:35:08.582793 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.582752 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zswrh" podStartSLOduration=18.193588606 podStartE2EDuration="33.582737565s" totalImagesPullingTime="15.389148959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:34:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:08.58158432 +0000 UTC m=+76.990006373" watchObservedRunningTime="2026-07-27 21:35:08.582737565 +0000 UTC m=+76.991159595" Jul 27 21:35:08.603389 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.603338 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tfdcz" podStartSLOduration=34.646955678 podStartE2EDuration="41.603321049s" totalImagesPullingTime="6.956365371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:08.602868064 +0000 UTC m=+77.011290094" watchObservedRunningTime="2026-07-27 21:35:08.603321049 +0000 UTC m=+77.011743080" Jul 27 21:35:08.644520 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.644463 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:08.647467 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.647416 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" podStartSLOduration=5.647396895 podStartE2EDuration="5.647396895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:08.629748873 +0000 UTC m=+77.038170907" watchObservedRunningTime="2026-07-27 21:35:08.647396895 +0000 UTC m=+77.055818924" Jul 27 21:35:08.647702 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.647555 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m24q9" podStartSLOduration=69.677033303 podStartE2EDuration="1m16.647547934s" totalImagesPullingTime="6.970514631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:08.646599787 +0000 UTC m=+77.055021818" watchObservedRunningTime="2026-07-27 21:35:08.647547934 +0000 UTC m=+77.055969966" Jul 27 21:35:08.744591 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:08.744554 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:09.703703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:09.703646 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:09.760289 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:09.760252 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:10.212799 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:10.212753 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:10.314283 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:10.314255 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:10.454492 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:10.454450 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:10.579202 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:10.579103 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" event={"ID":"790f9011-4695-4b70-8cf6-dcff8a0e6115","Type":"ContainerStarted","Data":"06bb9e6eea32be78cdc4cc847b343e0200b8cb9b5b4343a19ffbd0b2fd59617e"} Jul 27 21:35:10.597835 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:10.597782 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5655b9d984-xd8tk" podStartSLOduration=4.918960216 podStartE2EDuration="7.597767023s" totalImagesPullingTime="2.678806807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:10.596583565 +0000 UTC m=+79.005005595" watchObservedRunningTime="2026-07-27 21:35:10.597767023 +0000 UTC m=+79.006189054" Jul 27 21:35:12.112732 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:12.112335 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:12.589098 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:12.589059 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" event={"ID":"8cd11139-4a15-4a13-b231-aa7f3c1f27fc","Type":"ContainerStarted","Data":"a6abdca608d13960dae0a1bfec11b0e391dc2a689af8279b3cced915ac361d6c"} Jul 27 21:35:12.589270 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:12.589112 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" event={"ID":"8cd11139-4a15-4a13-b231-aa7f3c1f27fc","Type":"ContainerStarted","Data":"1fb8e36194e2fb8a8bfbe2adb61a977a1dc592b1072cdf5ca582455dd101b681"} Jul 27 21:35:12.589270 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:12.589130 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" event={"ID":"8cd11139-4a15-4a13-b231-aa7f3c1f27fc","Type":"ContainerStarted","Data":"84d84ad2776dcb6118dd98eafed589080903628fe95cb3d92b502bfc81521b10"} Jul 27 21:35:12.611857 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:12.611798 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6996759c8f-gg76s" podStartSLOduration=4.900541945 podStartE2EDuration="9.611778427s" totalImagesPullingTime="4.711236482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:12.610870749 +0000 UTC m=+81.019292780" watchObservedRunningTime="2026-07-27 21:35:12.611778427 +0000 UTC m=+81.020200458" Jul 27 21:35:13.501240 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:13.501204 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:13.564431 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:13.564394 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:14.739263 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:14.739221 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:14.843194 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:14.843156 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:14.943235 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:14.943192 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:17.379375 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:17.379338 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:17.447058 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:17.447025 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:18.372211 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:18.372174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:18.449068 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:18.449030 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:19.061431 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:19.061395 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:19.582171 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:19.582144 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tfdcz" Jul 27 21:35:19.707171 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:19.707136 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:19.790896 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:19.790862 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.022362 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.022320 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.223609 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.223572 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.330143 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.330054 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.399867 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.399841 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9f44c"] Jul 27 21:35:21.405081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.405060 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.409154 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.409118 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 21:35:21.409154 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.409155 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ghxlq\"" Jul 27 21:35:21.409360 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.409235 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 21:35:21.410128 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.410107 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 21:35:21.410315 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.410164 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 21:35:21.410315 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.410197 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 21:35:21.410415 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.410392 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 21:35:21.466158 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.466121 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.499125 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499087 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499125 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499124 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-tls\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499342 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499150 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-root\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499342 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-accelerators-collector-config\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499342 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499316 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-textfile\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499458 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499354 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-sys\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499458 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499381 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-wtmp\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499579 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499463 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7xcg\" (UniqueName: \"kubernetes.io/projected/4b1902dc-5464-4ef7-9e83-b6a74d10a929-kube-api-access-s7xcg\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.499579 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.499513 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b1902dc-5464-4ef7-9e83-b6a74d10a929-metrics-client-ca\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.557290 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.557256 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.600113 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600021 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-sys\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600113 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-wtmp\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600108 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7xcg\" (UniqueName: \"kubernetes.io/projected/4b1902dc-5464-4ef7-9e83-b6a74d10a929-kube-api-access-s7xcg\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600137 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-sys\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600160 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b1902dc-5464-4ef7-9e83-b6a74d10a929-metrics-client-ca\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600189 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600217 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-tls\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600252 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-root\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600269 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-wtmp\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600324 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b1902dc-5464-4ef7-9e83-b6a74d10a929-root\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600325 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-accelerators-collector-config\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600774 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-textfile\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600774 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600704 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-textfile\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600875 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600815 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b1902dc-5464-4ef7-9e83-b6a74d10a929-metrics-client-ca\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.600934 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.600912 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-accelerators-collector-config\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.603093 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.603061 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.603215 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.603135 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b1902dc-5464-4ef7-9e83-b6a74d10a929-node-exporter-tls\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.613007 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.612983 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7xcg\" (UniqueName: \"kubernetes.io/projected/4b1902dc-5464-4ef7-9e83-b6a74d10a929-kube-api-access-s7xcg\") pod \"node-exporter-9f44c\" (UID: \"4b1902dc-5464-4ef7-9e83-b6a74d10a929\") " pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.680930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.680895 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.714746 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.714712 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9f44c" Jul 27 21:35:21.726228 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:35:21.726181 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1902dc_5464_4ef7_9e83_b6a74d10a929.slice/crio-f6c18de5c50cb41be329b30634fdcdccbdb04da2e5eea06d10c196288bcdd4c1 WatchSource:0}: Error finding container f6c18de5c50cb41be329b30634fdcdccbdb04da2e5eea06d10c196288bcdd4c1: Status 404 returned error can't find the container with id f6c18de5c50cb41be329b30634fdcdccbdb04da2e5eea06d10c196288bcdd4c1 Jul 27 21:35:21.859888 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.859798 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:21.961684 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:21.961645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:22.101944 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:22.101905 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:22.205493 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:22.205450 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:22.305005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:22.304919 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:22.624761 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:22.624663 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f44c" event={"ID":"4b1902dc-5464-4ef7-9e83-b6a74d10a929","Type":"ContainerStarted","Data":"f6c18de5c50cb41be329b30634fdcdccbdb04da2e5eea06d10c196288bcdd4c1"} Jul 27 21:35:22.716986 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:22.716936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:23.512113 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:23.512086 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:35:23.633234 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:23.633194 2556 generic.go:356] "Generic (PLEG): container finished" podID="4b1902dc-5464-4ef7-9e83-b6a74d10a929" containerID="976c9ee9c6baf136c81f580863599b113f199c64789272ae93af865b263bce96" exitCode=0 Jul 27 21:35:23.633396 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:23.633249 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f44c" event={"ID":"4b1902dc-5464-4ef7-9e83-b6a74d10a929","Type":"ContainerDied","Data":"976c9ee9c6baf136c81f580863599b113f199c64789272ae93af865b263bce96"} Jul 27 21:35:24.236027 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:24.235987 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:24.321254 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:24.321217 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:24.389993 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:24.389958 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:24.638605 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:24.638510 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f44c" event={"ID":"4b1902dc-5464-4ef7-9e83-b6a74d10a929","Type":"ContainerStarted","Data":"d2de954cf85505c2018170b5dd5ad8ac00ae75ccafaa06a24572fdf6eae25af1"} Jul 27 21:35:24.638605 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:24.638571 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9f44c" event={"ID":"4b1902dc-5464-4ef7-9e83-b6a74d10a929","Type":"ContainerStarted","Data":"66dc733231537638bdd998cd6460909a2c52924f807ca95a62e022449716fa07"} Jul 27 21:35:25.755800 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:25.755759 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:25.848389 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:25.848337 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:25.941465 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:25.941429 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:26.287153 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:26.287106 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:26.370304 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:26.370262 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:27.284738 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:27.284704 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:27.385610 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:27.385577 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:27.634787 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:27.634689 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:28.530946 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.530900 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" podUID="69e19352-5ae6-454e-991e-1565c6672d0e" containerName="registry" containerID="cri-o://0e26665c367f16f7af6a799ca4eaffae9c50cae4b334b4240737204964d12da9" gracePeriod=30 Jul 27 21:35:28.653528 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.653500 2556 generic.go:356] "Generic (PLEG): container finished" podID="69e19352-5ae6-454e-991e-1565c6672d0e" containerID="0e26665c367f16f7af6a799ca4eaffae9c50cae4b334b4240737204964d12da9" exitCode=0 Jul 27 21:35:28.653657 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.653555 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" event={"ID":"69e19352-5ae6-454e-991e-1565c6672d0e","Type":"ContainerDied","Data":"0e26665c367f16f7af6a799ca4eaffae9c50cae4b334b4240737204964d12da9"} Jul 27 21:35:28.774630 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.774605 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:35:28.794974 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.794879 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9f44c" podStartSLOduration=6.68114767 podStartE2EDuration="7.794850527s" totalImagesPullingTime="1.043343786s" totalInitContainerRuntime="70.359071ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:24.662654179 +0000 UTC m=+93.071076208" watchObservedRunningTime="2026-07-27 21:35:28.794850527 +0000 UTC m=+97.203272554" Jul 27 21:35:28.856165 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856125 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-bound-sa-token\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856333 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856173 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-trusted-ca\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856333 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856198 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-image-registry-private-configuration\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856333 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856236 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69e19352-5ae6-454e-991e-1565c6672d0e-ca-trust-extracted\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856333 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856284 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856333 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856317 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-installation-pull-secrets\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856544 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856347 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjcpt\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-kube-api-access-cjcpt\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856544 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856372 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-registry-certificates\") pod \"69e19352-5ae6-454e-991e-1565c6672d0e\" (UID: \"69e19352-5ae6-454e-991e-1565c6672d0e\") " Jul 27 21:35:28.856725 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.856698 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:35:28.857168 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.857118 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:35:28.858934 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.858905 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:35:28.859064 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.858935 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:35:28.859146 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.859111 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:35:28.859235 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.859213 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-kube-api-access-cjcpt" (OuterVolumeSpecName: "kube-api-access-cjcpt") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "kube-api-access-cjcpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:35:28.859272 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.859242 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:35:28.868251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.868213 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e19352-5ae6-454e-991e-1565c6672d0e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "69e19352-5ae6-454e-991e-1565c6672d0e" (UID: "69e19352-5ae6-454e-991e-1565c6672d0e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:35:28.957852 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957810 2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-bound-sa-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:28.957852 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957847 2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-trusted-ca\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:28.958050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957861 2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-image-registry-private-configuration\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:28.958050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957875 2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/69e19352-5ae6-454e-991e-1565c6672d0e-ca-trust-extracted\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:28.958050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957887 2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-registry-tls\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:28.958050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957899 2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/69e19352-5ae6-454e-991e-1565c6672d0e-installation-pull-secrets\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:28.958050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957910 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjcpt\" (UniqueName: \"kubernetes.io/projected/69e19352-5ae6-454e-991e-1565c6672d0e-kube-api-access-cjcpt\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:28.958050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:28.957922 2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/69e19352-5ae6-454e-991e-1565c6672d0e-registry-certificates\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:35:29.184215 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.184131 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:29.285188 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.285156 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:29.585163 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.585130 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:29.657662 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.657633 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" Jul 27 21:35:29.657662 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.657640 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-777dc78d4c-rndfg" event={"ID":"69e19352-5ae6-454e-991e-1565c6672d0e","Type":"ContainerDied","Data":"20b7a42156824b11c804e0bca03a8e9a3ae4f5df1eefde91481ecea6f7f2f939"} Jul 27 21:35:29.657854 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.657733 2556 scope.go:117] "RemoveContainer" containerID="0e26665c367f16f7af6a799ca4eaffae9c50cae4b334b4240737204964d12da9" Jul 27 21:35:29.679937 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.679912 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-777dc78d4c-rndfg"] Jul 27 21:35:29.683802 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.683776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:29.684061 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.684045 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-777dc78d4c-rndfg"] Jul 27 21:35:29.904999 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.904919 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:29.965494 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:29.965458 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:30.190796 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:30.190720 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e19352-5ae6-454e-991e-1565c6672d0e" path="/var/lib/kubelet/pods/69e19352-5ae6-454e-991e-1565c6672d0e/volumes" Jul 27 21:35:30.302044 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:30.302010 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:30.370979 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:30.370940 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:30.583411 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:30.583384 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-797cb7d859-wmrhq" Jul 27 21:35:30.731110 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:30.731073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:30.832088 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:30.832052 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:31.069251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:31.069219 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:31.166538 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:31.166505 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:36.298982 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:36.298945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:36.398789 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:36.398749 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:38.708525 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:38.708485 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:38.808323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:38.808287 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:39.576573 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:39.576546 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-85djc" Jul 27 21:35:42.027074 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:42.027028 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:42.123146 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:42.123110 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:42.478698 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:42.478652 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:42.582813 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:42.582773 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:43.026408 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:43.026371 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:49.702118 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:49.702079 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:35:49.804190 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:35:49.804140 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:15.362460 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:15.362414 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:21.992747 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:21.992708 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:22.086047 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:22.086011 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:37.529093 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:37.529053 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:37.636375 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:37.636334 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:41.757132 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:41.757093 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:41.816721 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:41.816685 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:49.515985 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:49.515943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:49.580074 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:49.580037 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:52.769592 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:52.769556 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:52.865475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:52.865437 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:36:52.937694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:36:52.937645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:37:04.708540 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:04.708504 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:37:04.809145 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:04.809102 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:37:36.493320 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:36.493282 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:37:36.555158 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:36.555117 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:37:48.876227 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:48.876186 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:37:48.972207 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:48.972173 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:37:49.617340 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.617309 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j"] Jul 27 21:37:49.617553 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.617540 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e19352-5ae6-454e-991e-1565c6672d0e" containerName="registry" Jul 27 21:37:49.617553 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.617553 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e19352-5ae6-454e-991e-1565c6672d0e" containerName="registry" Jul 27 21:37:49.617633 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.617595 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e19352-5ae6-454e-991e-1565c6672d0e" containerName="registry" Jul 27 21:37:49.619446 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.619430 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.622624 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.622603 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:37:49.622961 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.622946 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:37:49.622961 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.622955 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bdqdb\"" Jul 27 21:37:49.631491 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.631464 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j"] Jul 27 21:37:49.679992 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.679950 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdppx\" (UniqueName: \"kubernetes.io/projected/752177b9-2c11-4aa3-8f84-ac697ba383b7-kube-api-access-pdppx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.680173 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.680000 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.680173 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.680062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.781227 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.781193 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdppx\" (UniqueName: \"kubernetes.io/projected/752177b9-2c11-4aa3-8f84-ac697ba383b7-kube-api-access-pdppx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.781377 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.781236 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.781377 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.781259 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.781592 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.781575 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.781662 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.781643 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.789930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.789907 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdppx\" (UniqueName: \"kubernetes.io/projected/752177b9-2c11-4aa3-8f84-ac697ba383b7-kube-api-access-pdppx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:49.928359 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:49.928249 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:37:50.094255 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:50.094233 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j"] Jul 27 21:37:50.096783 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:37:50.096754 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752177b9_2c11_4aa3_8f84_ac697ba383b7.slice/crio-898790c8e8a5ce6dc5f0214505388d2080d4af4b9b133dd89dc097249e0df511 WatchSource:0}: Error finding container 898790c8e8a5ce6dc5f0214505388d2080d4af4b9b133dd89dc097249e0df511: Status 404 returned error can't find the container with id 898790c8e8a5ce6dc5f0214505388d2080d4af4b9b133dd89dc097249e0df511 Jul 27 21:37:50.136733 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:50.136696 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" event={"ID":"752177b9-2c11-4aa3-8f84-ac697ba383b7","Type":"ContainerStarted","Data":"898790c8e8a5ce6dc5f0214505388d2080d4af4b9b133dd89dc097249e0df511"} Jul 27 21:37:57.163310 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:57.163273 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" event={"ID":"752177b9-2c11-4aa3-8f84-ac697ba383b7","Type":"ContainerStarted","Data":"eafa3ef10794df2288a9e0c094f51b45bbd99975ae9fc45a0548458507cabe05"} Jul 27 21:37:58.168579 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:58.168543 2556 generic.go:356] "Generic (PLEG): container finished" podID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerID="eafa3ef10794df2288a9e0c094f51b45bbd99975ae9fc45a0548458507cabe05" exitCode=0 Jul 27 21:37:58.169007 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:37:58.168607 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" event={"ID":"752177b9-2c11-4aa3-8f84-ac697ba383b7","Type":"ContainerDied","Data":"eafa3ef10794df2288a9e0c094f51b45bbd99975ae9fc45a0548458507cabe05"} Jul 27 21:38:01.181403 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:01.181370 2556 generic.go:356] "Generic (PLEG): container finished" podID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerID="2382dee8ff328cbea36f407113757608666b87a7c1ab2702e0f5030d2bc386e3" exitCode=0 Jul 27 21:38:01.181828 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:01.181422 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" event={"ID":"752177b9-2c11-4aa3-8f84-ac697ba383b7","Type":"ContainerDied","Data":"2382dee8ff328cbea36f407113757608666b87a7c1ab2702e0f5030d2bc386e3"} Jul 27 21:38:09.211276 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:09.211242 2556 generic.go:356] "Generic (PLEG): container finished" podID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerID="d91e1a5e312d9fe2cc4fb5c1289fbe4b80e7efe8d30573750a396f694105712a" exitCode=0 Jul 27 21:38:09.211689 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:09.211301 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" event={"ID":"752177b9-2c11-4aa3-8f84-ac697ba383b7","Type":"ContainerDied","Data":"d91e1a5e312d9fe2cc4fb5c1289fbe4b80e7efe8d30573750a396f694105712a"} Jul 27 21:38:10.338775 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.338739 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:38:10.444290 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.444245 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdppx\" (UniqueName: \"kubernetes.io/projected/752177b9-2c11-4aa3-8f84-ac697ba383b7-kube-api-access-pdppx\") pod \"752177b9-2c11-4aa3-8f84-ac697ba383b7\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " Jul 27 21:38:10.444493 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.444344 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-util\") pod \"752177b9-2c11-4aa3-8f84-ac697ba383b7\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " Jul 27 21:38:10.444493 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.444398 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-bundle\") pod \"752177b9-2c11-4aa3-8f84-ac697ba383b7\" (UID: \"752177b9-2c11-4aa3-8f84-ac697ba383b7\") " Jul 27 21:38:10.445020 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.444996 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-bundle" (OuterVolumeSpecName: "bundle") pod "752177b9-2c11-4aa3-8f84-ac697ba383b7" (UID: "752177b9-2c11-4aa3-8f84-ac697ba383b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:38:10.446660 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.446638 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752177b9-2c11-4aa3-8f84-ac697ba383b7-kube-api-access-pdppx" (OuterVolumeSpecName: "kube-api-access-pdppx") pod "752177b9-2c11-4aa3-8f84-ac697ba383b7" (UID: "752177b9-2c11-4aa3-8f84-ac697ba383b7"). InnerVolumeSpecName "kube-api-access-pdppx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:38:10.448838 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.448813 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-util" (OuterVolumeSpecName: "util") pod "752177b9-2c11-4aa3-8f84-ac697ba383b7" (UID: "752177b9-2c11-4aa3-8f84-ac697ba383b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:38:10.545771 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.545687 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:38:10.545771 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.545712 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752177b9-2c11-4aa3-8f84-ac697ba383b7-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:38:10.545771 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:10.545721 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdppx\" (UniqueName: \"kubernetes.io/projected/752177b9-2c11-4aa3-8f84-ac697ba383b7-kube-api-access-pdppx\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:38:11.218966 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:11.218927 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" event={"ID":"752177b9-2c11-4aa3-8f84-ac697ba383b7","Type":"ContainerDied","Data":"898790c8e8a5ce6dc5f0214505388d2080d4af4b9b133dd89dc097249e0df511"} Jul 27 21:38:11.219138 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:11.218980 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jn42j" Jul 27 21:38:11.219138 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:11.218982 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898790c8e8a5ce6dc5f0214505388d2080d4af4b9b133dd89dc097249e0df511" Jul 27 21:38:15.831303 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:15.831211 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:16.606761 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:16.606722 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:16.696464 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:16.696425 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:17.762073 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762025 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7"] Jul 27 21:38:17.762475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762269 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerName="pull" Jul 27 21:38:17.762475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762279 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerName="pull" Jul 27 21:38:17.762475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762292 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerName="extract" Jul 27 21:38:17.762475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762297 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerName="extract" Jul 27 21:38:17.762475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762309 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerName="util" Jul 27 21:38:17.762475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762315 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerName="util" Jul 27 21:38:17.762475 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.762356 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="752177b9-2c11-4aa3-8f84-ac697ba383b7" containerName="extract" Jul 27 21:38:17.765770 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.765750 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:17.768694 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.768662 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xk47k\"" Jul 27 21:38:17.768972 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.768955 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:38:17.769573 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.769554 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 27 21:38:17.779605 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.779570 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7"] Jul 27 21:38:17.896995 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.896940 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88w9w\" (UniqueName: \"kubernetes.io/projected/df994aa2-ad2f-468c-adb8-e286d9035de3-kube-api-access-88w9w\") pod \"cert-manager-operator-controller-manager-55c444bf86-87rc7\" (UID: \"df994aa2-ad2f-468c-adb8-e286d9035de3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:17.896995 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.896994 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df994aa2-ad2f-468c-adb8-e286d9035de3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-87rc7\" (UID: \"df994aa2-ad2f-468c-adb8-e286d9035de3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:17.997848 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.997817 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88w9w\" (UniqueName: \"kubernetes.io/projected/df994aa2-ad2f-468c-adb8-e286d9035de3-kube-api-access-88w9w\") pod \"cert-manager-operator-controller-manager-55c444bf86-87rc7\" (UID: \"df994aa2-ad2f-468c-adb8-e286d9035de3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:17.997949 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.997853 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df994aa2-ad2f-468c-adb8-e286d9035de3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-87rc7\" (UID: \"df994aa2-ad2f-468c-adb8-e286d9035de3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:17.998184 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:17.998165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df994aa2-ad2f-468c-adb8-e286d9035de3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-87rc7\" (UID: \"df994aa2-ad2f-468c-adb8-e286d9035de3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:18.010330 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:18.010299 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88w9w\" (UniqueName: \"kubernetes.io/projected/df994aa2-ad2f-468c-adb8-e286d9035de3-kube-api-access-88w9w\") pod \"cert-manager-operator-controller-manager-55c444bf86-87rc7\" (UID: \"df994aa2-ad2f-468c-adb8-e286d9035de3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:18.074256 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:18.074160 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" Jul 27 21:38:18.215368 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:18.215331 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7"] Jul 27 21:38:18.219240 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:38:18.219216 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf994aa2_ad2f_468c_adb8_e286d9035de3.slice/crio-60e600ed13f7f70a19184d60ae1cf3a752c27fbad74533cd4cd145628a090c71 WatchSource:0}: Error finding container 60e600ed13f7f70a19184d60ae1cf3a752c27fbad74533cd4cd145628a090c71: Status 404 returned error can't find the container with id 60e600ed13f7f70a19184d60ae1cf3a752c27fbad74533cd4cd145628a090c71 Jul 27 21:38:18.241818 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:18.241789 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" event={"ID":"df994aa2-ad2f-468c-adb8-e286d9035de3","Type":"ContainerStarted","Data":"60e600ed13f7f70a19184d60ae1cf3a752c27fbad74533cd4cd145628a090c71"} Jul 27 21:38:19.411409 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:19.411371 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:19.512005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:19.511960 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:20.251264 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:20.251229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" event={"ID":"df994aa2-ad2f-468c-adb8-e286d9035de3","Type":"ContainerStarted","Data":"e45e9458fc626351fd3e84af428600ef1b1ab8dd11d625665b8010e6cd7272bb"} Jul 27 21:38:20.278650 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:20.278589 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87rc7" podStartSLOduration=1.6022464589999998 podStartE2EDuration="3.278567237s" totalImagesPullingTime="1.676320778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:38:20.276330606 +0000 UTC m=+268.684752650" watchObservedRunningTime="2026-07-27 21:38:20.278567237 +0000 UTC m=+268.686989269" Jul 27 21:38:20.794824 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:20.794787 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:21.029193 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:21.029155 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:21.188411 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:21.188293 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:23.286851 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:23.286813 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:23.502145 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:23.502115 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:26.211765 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:26.211734 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:26.311563 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:26.311525 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:30.949600 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:30.949546 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:33.095425 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:33.095392 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:44.137157 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:44.137112 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:44.236887 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:44.236848 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:45.878480 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.878444 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p"] Jul 27 21:38:45.880526 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.880510 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:45.883307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.883281 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 27 21:38:45.884233 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.884218 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9sp6d\"" Jul 27 21:38:45.884318 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.884267 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 27 21:38:45.895092 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.895068 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p"] Jul 27 21:38:45.908206 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.908170 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afaa0467-831b-4304-b129-d4139f622133-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cn56p\" (UID: \"afaa0467-831b-4304-b129-d4139f622133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:45.908206 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:45.908204 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrnw\" (UniqueName: \"kubernetes.io/projected/afaa0467-831b-4304-b129-d4139f622133-kube-api-access-czrnw\") pod \"openshift-lws-operator-fd8ccff4c-cn56p\" (UID: \"afaa0467-831b-4304-b129-d4139f622133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:46.009459 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:46.009422 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afaa0467-831b-4304-b129-d4139f622133-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cn56p\" (UID: \"afaa0467-831b-4304-b129-d4139f622133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:46.009459 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:46.009458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrnw\" (UniqueName: \"kubernetes.io/projected/afaa0467-831b-4304-b129-d4139f622133-kube-api-access-czrnw\") pod \"openshift-lws-operator-fd8ccff4c-cn56p\" (UID: \"afaa0467-831b-4304-b129-d4139f622133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:46.009837 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:46.009816 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afaa0467-831b-4304-b129-d4139f622133-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cn56p\" (UID: \"afaa0467-831b-4304-b129-d4139f622133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:46.018957 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:46.018930 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrnw\" (UniqueName: \"kubernetes.io/projected/afaa0467-831b-4304-b129-d4139f622133-kube-api-access-czrnw\") pod \"openshift-lws-operator-fd8ccff4c-cn56p\" (UID: \"afaa0467-831b-4304-b129-d4139f622133\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:46.188883 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:46.188810 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" Jul 27 21:38:46.316111 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:46.316080 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p"] Jul 27 21:38:46.319204 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:38:46.319176 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafaa0467_831b_4304_b129_d4139f622133.slice/crio-dfc30295f1e8d79221027d08dcc6b347cefad8fea55a00244162f4a4de4d44df WatchSource:0}: Error finding container dfc30295f1e8d79221027d08dcc6b347cefad8fea55a00244162f4a4de4d44df: Status 404 returned error can't find the container with id dfc30295f1e8d79221027d08dcc6b347cefad8fea55a00244162f4a4de4d44df Jul 27 21:38:46.342361 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:46.342326 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" event={"ID":"afaa0467-831b-4304-b129-d4139f622133","Type":"ContainerStarted","Data":"dfc30295f1e8d79221027d08dcc6b347cefad8fea55a00244162f4a4de4d44df"} Jul 27 21:38:49.356042 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:49.356008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" event={"ID":"afaa0467-831b-4304-b129-d4139f622133","Type":"ContainerStarted","Data":"1834c4a716f24edc33e5cf5eff68fd6f9a903d29bc54298d676e1333470b498f"} Jul 27 21:38:49.372553 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:49.372504 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cn56p" podStartSLOduration=1.9477586599999999 podStartE2EDuration="4.372484353s" totalImagesPullingTime="2.424725693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:38:49.371607464 +0000 UTC m=+297.780029493" watchObservedRunningTime="2026-07-27 21:38:49.372484353 +0000 UTC m=+297.780906386" Jul 27 21:38:50.151094 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:50.151060 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:50.251727 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:50.251662 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:51.587131 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.587097 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk"] Jul 27 21:38:51.589424 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.589407 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.592000 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.591975 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 21:38:51.592129 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.591977 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 21:38:51.593140 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.593119 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bdqdb\"" Jul 27 21:38:51.598915 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.598892 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk"] Jul 27 21:38:51.654391 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.654361 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.654391 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.654401 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f49x\" (UniqueName: \"kubernetes.io/projected/557dedbb-f0bc-4c42-b987-aa59c7b478a4-kube-api-access-5f49x\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.654639 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.654499 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.755176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.755135 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.755356 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.755186 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.755356 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.755221 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f49x\" (UniqueName: \"kubernetes.io/projected/557dedbb-f0bc-4c42-b987-aa59c7b478a4-kube-api-access-5f49x\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.755569 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.755549 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.755707 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.755660 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.763542 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.763509 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f49x\" (UniqueName: \"kubernetes.io/projected/557dedbb-f0bc-4c42-b987-aa59c7b478a4-kube-api-access-5f49x\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.831588 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.831546 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:51.899867 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.899773 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:51.986650 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:51.986609 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 21:38:52.043462 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:52.043422 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk"] Jul 27 21:38:52.048391 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:38:52.048359 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557dedbb_f0bc_4c42_b987_aa59c7b478a4.slice/crio-9fcac91fe115f24cdfbfc3b8e5594548e36cd85ac30e3ad7d1750f4d06fae6ff WatchSource:0}: Error finding container 9fcac91fe115f24cdfbfc3b8e5594548e36cd85ac30e3ad7d1750f4d06fae6ff: Status 404 returned error can't find the container with id 9fcac91fe115f24cdfbfc3b8e5594548e36cd85ac30e3ad7d1750f4d06fae6ff Jul 27 21:38:52.368898 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:52.368866 2556 generic.go:356] "Generic (PLEG): container finished" podID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerID="975b26b4685cbe7d784f156153762c9879905a60198ee326249d96a9592ec54f" exitCode=0 Jul 27 21:38:52.369108 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:52.368958 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" event={"ID":"557dedbb-f0bc-4c42-b987-aa59c7b478a4","Type":"ContainerDied","Data":"975b26b4685cbe7d784f156153762c9879905a60198ee326249d96a9592ec54f"} Jul 27 21:38:52.369108 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:52.369011 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" event={"ID":"557dedbb-f0bc-4c42-b987-aa59c7b478a4","Type":"ContainerStarted","Data":"9fcac91fe115f24cdfbfc3b8e5594548e36cd85ac30e3ad7d1750f4d06fae6ff"} Jul 27 21:38:52.617600 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:52.617547 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:53.374661 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:53.374580 2556 generic.go:356] "Generic (PLEG): container finished" podID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerID="2d15fa0c39de3d0cafdf585944ba3e00ed6c58a9ce6b818afe0b0912e385da00" exitCode=0 Jul 27 21:38:53.374833 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:53.374681 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" event={"ID":"557dedbb-f0bc-4c42-b987-aa59c7b478a4","Type":"ContainerDied","Data":"2d15fa0c39de3d0cafdf585944ba3e00ed6c58a9ce6b818afe0b0912e385da00"} Jul 27 21:38:54.062274 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:54.062241 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:54.380836 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:54.380749 2556 generic.go:356] "Generic (PLEG): container finished" podID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerID="bd0347e7a0cea2926b06ac45eb99777f4548b1814f4ce0729f22e0c978b50276" exitCode=0 Jul 27 21:38:54.380836 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:54.380807 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" event={"ID":"557dedbb-f0bc-4c42-b987-aa59c7b478a4","Type":"ContainerDied","Data":"bd0347e7a0cea2926b06ac45eb99777f4548b1814f4ce0729f22e0c978b50276"} Jul 27 21:38:55.512902 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.512877 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:55.587241 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.587209 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f49x\" (UniqueName: \"kubernetes.io/projected/557dedbb-f0bc-4c42-b987-aa59c7b478a4-kube-api-access-5f49x\") pod \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " Jul 27 21:38:55.587241 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.587257 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-util\") pod \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " Jul 27 21:38:55.587527 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.587313 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-bundle\") pod \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\" (UID: \"557dedbb-f0bc-4c42-b987-aa59c7b478a4\") " Jul 27 21:38:55.588105 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.588077 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-bundle" (OuterVolumeSpecName: "bundle") pod "557dedbb-f0bc-4c42-b987-aa59c7b478a4" (UID: "557dedbb-f0bc-4c42-b987-aa59c7b478a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:38:55.589492 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.589468 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557dedbb-f0bc-4c42-b987-aa59c7b478a4-kube-api-access-5f49x" (OuterVolumeSpecName: "kube-api-access-5f49x") pod "557dedbb-f0bc-4c42-b987-aa59c7b478a4" (UID: "557dedbb-f0bc-4c42-b987-aa59c7b478a4"). InnerVolumeSpecName "kube-api-access-5f49x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:38:55.593322 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.593257 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-util" (OuterVolumeSpecName: "util") pod "557dedbb-f0bc-4c42-b987-aa59c7b478a4" (UID: "557dedbb-f0bc-4c42-b987-aa59c7b478a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:38:55.688641 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.688551 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:38:55.688641 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.688580 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f49x\" (UniqueName: \"kubernetes.io/projected/557dedbb-f0bc-4c42-b987-aa59c7b478a4-kube-api-access-5f49x\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:38:55.688641 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:55.688591 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/557dedbb-f0bc-4c42-b987-aa59c7b478a4-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:38:56.390928 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:56.390902 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" Jul 27 21:38:56.391093 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:56.390902 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdftdk" event={"ID":"557dedbb-f0bc-4c42-b987-aa59c7b478a4","Type":"ContainerDied","Data":"9fcac91fe115f24cdfbfc3b8e5594548e36cd85ac30e3ad7d1750f4d06fae6ff"} Jul 27 21:38:56.391093 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:56.391022 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fcac91fe115f24cdfbfc3b8e5594548e36cd85ac30e3ad7d1750f4d06fae6ff" Jul 27 21:38:58.050329 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:58.050300 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:38:58.151645 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:38:58.151606 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:04.219874 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:04.219841 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:04.371632 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:04.371589 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:04.823129 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:04.823095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:04.982744 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:04.982709 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:05.598302 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:05.598257 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:05.980827 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:05.980788 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.078500 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.078457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.115954 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.115925 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46"] Jul 27 21:39:06.116218 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.116204 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerName="util" Jul 27 21:39:06.116255 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.116220 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerName="util" Jul 27 21:39:06.116255 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.116232 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerName="pull" Jul 27 21:39:06.116255 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.116238 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerName="pull" Jul 27 21:39:06.116255 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.116245 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerName="extract" Jul 27 21:39:06.116255 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.116250 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerName="extract" Jul 27 21:39:06.116401 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.116288 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="557dedbb-f0bc-4c42-b987-aa59c7b478a4" containerName="extract" Jul 27 21:39:06.120370 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.120354 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.123585 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.123567 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 21:39:06.123919 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.123903 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 21:39:06.124138 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.124122 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 21:39:06.124213 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.124198 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 21:39:06.124286 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.124271 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xl2cn\"" Jul 27 21:39:06.142211 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.142184 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46"] Jul 27 21:39:06.163928 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.163902 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflr8\" (UniqueName: \"kubernetes.io/projected/83de5dde-f09f-438f-8dc8-74889a1429ae-kube-api-access-gflr8\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.164075 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.163942 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83de5dde-f09f-438f-8dc8-74889a1429ae-manifests\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.164075 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.163972 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83de5dde-f09f-438f-8dc8-74889a1429ae-tmp\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.164075 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.164004 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/83de5dde-f09f-438f-8dc8-74889a1429ae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.164181 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.164077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83de5dde-f09f-438f-8dc8-74889a1429ae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.264753 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.264636 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83de5dde-f09f-438f-8dc8-74889a1429ae-tmp\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.264753 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.264710 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/83de5dde-f09f-438f-8dc8-74889a1429ae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.264753 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.264749 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83de5dde-f09f-438f-8dc8-74889a1429ae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.264996 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.264803 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflr8\" (UniqueName: \"kubernetes.io/projected/83de5dde-f09f-438f-8dc8-74889a1429ae-kube-api-access-gflr8\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.264996 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.264830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83de5dde-f09f-438f-8dc8-74889a1429ae-manifests\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.265216 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.265178 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/83de5dde-f09f-438f-8dc8-74889a1429ae-manifests\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.267289 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.267260 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83de5dde-f09f-438f-8dc8-74889a1429ae-tmp\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.267417 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.267400 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/83de5dde-f09f-438f-8dc8-74889a1429ae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.267470 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.267396 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83de5dde-f09f-438f-8dc8-74889a1429ae-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.281281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.281249 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflr8\" (UniqueName: \"kubernetes.io/projected/83de5dde-f09f-438f-8dc8-74889a1429ae-kube-api-access-gflr8\") pod \"opendatahub-operator-controller-manager-5d57c545b5-57d46\" (UID: \"83de5dde-f09f-438f-8dc8-74889a1429ae\") " pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.282458 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.282428 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.390730 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.390662 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.430637 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.430606 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:06.526344 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.526258 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.594615 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.594595 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46"] Jul 27 21:39:06.597308 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:06.597283 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83de5dde_f09f_438f_8dc8_74889a1429ae.slice/crio-12f0d15acca8acc5282805ee5f28f7cd8d7339266d6c504e3574c50d534f993f WatchSource:0}: Error finding container 12f0d15acca8acc5282805ee5f28f7cd8d7339266d6c504e3574c50d534f993f: Status 404 returned error can't find the container with id 12f0d15acca8acc5282805ee5f28f7cd8d7339266d6c504e3574c50d534f993f Jul 27 21:39:06.599103 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.599085 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:39:06.603380 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.603346 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.681938 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.681906 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.778172 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.778077 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:06.877548 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:06.877514 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:07.439845 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:07.439811 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" event={"ID":"83de5dde-f09f-438f-8dc8-74889a1429ae","Type":"ContainerStarted","Data":"12f0d15acca8acc5282805ee5f28f7cd8d7339266d6c504e3574c50d534f993f"} Jul 27 21:39:10.454833 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:10.454798 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" event={"ID":"83de5dde-f09f-438f-8dc8-74889a1429ae","Type":"ContainerStarted","Data":"36b6782c6819978e54b2f3f252850e011e72f29d5a7301b374db63217ac1e971"} Jul 27 21:39:11.460390 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:11.460350 2556 generic.go:356] "Generic (PLEG): container finished" podID="83de5dde-f09f-438f-8dc8-74889a1429ae" containerID="36b6782c6819978e54b2f3f252850e011e72f29d5a7301b374db63217ac1e971" exitCode=0 Jul 27 21:39:11.460795 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:11.460419 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" event={"ID":"83de5dde-f09f-438f-8dc8-74889a1429ae","Type":"ContainerDied","Data":"36b6782c6819978e54b2f3f252850e011e72f29d5a7301b374db63217ac1e971"} Jul 27 21:39:12.466615 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:12.466567 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" event={"ID":"83de5dde-f09f-438f-8dc8-74889a1429ae","Type":"ContainerStarted","Data":"e070e101c7b33bbb314c2f88a713ab299975a0f1296a127e7a52f5d27b39bdbd"} Jul 27 21:39:12.467005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:12.466712 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:12.489225 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:12.489169 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" podStartSLOduration=1.9134992 podStartE2EDuration="6.489152451s" totalImagesPullingTime="3.53036289s" totalInitContainerRuntime="1.045290361s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:12.487487132 +0000 UTC m=+320.895909173" watchObservedRunningTime="2026-07-27 21:39:12.489152451 +0000 UTC m=+320.897574481" Jul 27 21:39:14.604468 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:14.604424 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:16.351191 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:16.351158 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:16.411884 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:16.411825 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:20.646763 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:20.646727 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:20.747364 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:20.747327 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:21.090440 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:21.090405 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:21.154136 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:21.154099 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:23.472914 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:23.472886 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d57c545b5-57d46" Jul 27 21:39:25.363778 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:25.363742 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:25.497899 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:25.497849 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:28.472994 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.472942 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:28.616158 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.616124 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx"] Jul 27 21:39:28.623629 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.623603 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.626974 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.626948 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 21:39:28.627097 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.626976 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 21:39:28.627708 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.627695 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 21:39:28.627936 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.627924 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 21:39:28.628010 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.627995 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-klfc9\"" Jul 27 21:39:28.656968 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.656940 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx"] Jul 27 21:39:28.657765 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.657736 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:28.743039 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.742947 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a92de9c-6da0-41be-908f-ba23b2de00ba-tls-certs\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.743039 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.743013 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a92de9c-6da0-41be-908f-ba23b2de00ba-tmp\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.743246 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.743058 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcgk\" (UniqueName: \"kubernetes.io/projected/4a92de9c-6da0-41be-908f-ba23b2de00ba-kube-api-access-8qcgk\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.812389 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.812353 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qhx8r"] Jul 27 21:39:28.814578 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.814556 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:28.819941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.819917 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q7hng\"" Jul 27 21:39:28.820057 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.820019 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 21:39:28.834395 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.834370 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qhx8r"] Jul 27 21:39:28.844220 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.844192 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a92de9c-6da0-41be-908f-ba23b2de00ba-tmp\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.844311 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.844246 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcgk\" (UniqueName: \"kubernetes.io/projected/4a92de9c-6da0-41be-908f-ba23b2de00ba-kube-api-access-8qcgk\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.844311 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.844297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a92de9c-6da0-41be-908f-ba23b2de00ba-tls-certs\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.846540 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.846511 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a92de9c-6da0-41be-908f-ba23b2de00ba-tmp\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.846792 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.846776 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a92de9c-6da0-41be-908f-ba23b2de00ba-tls-certs\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.853726 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.853704 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcgk\" (UniqueName: \"kubernetes.io/projected/4a92de9c-6da0-41be-908f-ba23b2de00ba-kube-api-access-8qcgk\") pod \"kube-auth-proxy-6fd65f9cdf-8tcdx\" (UID: \"4a92de9c-6da0-41be-908f-ba23b2de00ba\") " pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.862276 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.862246 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:28.933111 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.933081 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" Jul 27 21:39:28.945539 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.945491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrj5\" (UniqueName: \"kubernetes.io/projected/5ca527b1-69df-4373-94cc-e0a164209688-kube-api-access-gcrj5\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:28.945691 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.945565 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:28.966140 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:28.966102 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:29.046226 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.046195 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:29.046388 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.046252 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrj5\" (UniqueName: \"kubernetes.io/projected/5ca527b1-69df-4373-94cc-e0a164209688-kube-api-access-gcrj5\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:29.046432 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:39:29.046378 2556 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 27 21:39:29.046466 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:39:29.046460 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert podName:5ca527b1-69df-4373-94cc-e0a164209688 nodeName:}" failed. No retries permitted until 2026-07-27 21:39:29.546437203 +0000 UTC m=+337.954859213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert") pod "odh-model-controller-7998c67dc4-qhx8r" (UID: "5ca527b1-69df-4373-94cc-e0a164209688") : secret "odh-model-controller-webhook-cert" not found Jul 27 21:39:29.057735 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.057705 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrj5\" (UniqueName: \"kubernetes.io/projected/5ca527b1-69df-4373-94cc-e0a164209688-kube-api-access-gcrj5\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:29.064046 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.064010 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:29.072307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.072287 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx"] Jul 27 21:39:29.075318 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:29.075290 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a92de9c_6da0_41be_908f_ba23b2de00ba.slice/crio-7ee4fc48758312ebe35ec3db5d9e2dacbd31a2156b55b7918ba4b3277480c282 WatchSource:0}: Error finding container 7ee4fc48758312ebe35ec3db5d9e2dacbd31a2156b55b7918ba4b3277480c282: Status 404 returned error can't find the container with id 7ee4fc48758312ebe35ec3db5d9e2dacbd31a2156b55b7918ba4b3277480c282 Jul 27 21:39:29.537490 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.537453 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" event={"ID":"4a92de9c-6da0-41be-908f-ba23b2de00ba","Type":"ContainerStarted","Data":"7ee4fc48758312ebe35ec3db5d9e2dacbd31a2156b55b7918ba4b3277480c282"} Jul 27 21:39:29.551839 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.551804 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:29.552008 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:39:29.551970 2556 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 27 21:39:29.552067 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:39:29.552056 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert podName:5ca527b1-69df-4373-94cc-e0a164209688 nodeName:}" failed. No retries permitted until 2026-07-27 21:39:30.552034617 +0000 UTC m=+338.960456628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert") pod "odh-model-controller-7998c67dc4-qhx8r" (UID: "5ca527b1-69df-4373-94cc-e0a164209688") : secret "odh-model-controller-webhook-cert" not found Jul 27 21:39:29.566064 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.566018 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vqssk"] Jul 27 21:39:29.568645 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.568626 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:29.572727 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.572707 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 27 21:39:29.585731 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.585707 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vqssk"] Jul 27 21:39:29.605127 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.605090 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:29.652481 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.652447 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cfb657-58c7-4501-aa5b-4883ee907a5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-vqssk\" (UID: \"f3cfb657-58c7-4501-aa5b-4883ee907a5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:29.652785 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.652504 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n52jd\" (UniqueName: \"kubernetes.io/projected/f3cfb657-58c7-4501-aa5b-4883ee907a5e-kube-api-access-n52jd\") pod \"model-serving-api-54d89bd5fc-vqssk\" (UID: \"f3cfb657-58c7-4501-aa5b-4883ee907a5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:29.753881 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.753845 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cfb657-58c7-4501-aa5b-4883ee907a5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-vqssk\" (UID: \"f3cfb657-58c7-4501-aa5b-4883ee907a5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:29.754089 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.753909 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n52jd\" (UniqueName: \"kubernetes.io/projected/f3cfb657-58c7-4501-aa5b-4883ee907a5e-kube-api-access-n52jd\") pod \"model-serving-api-54d89bd5fc-vqssk\" (UID: \"f3cfb657-58c7-4501-aa5b-4883ee907a5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:29.754089 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:39:29.754017 2556 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 27 21:39:29.754209 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:39:29.754100 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3cfb657-58c7-4501-aa5b-4883ee907a5e-tls-certs podName:f3cfb657-58c7-4501-aa5b-4883ee907a5e nodeName:}" failed. No retries permitted until 2026-07-27 21:39:30.254079525 +0000 UTC m=+338.662501548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f3cfb657-58c7-4501-aa5b-4883ee907a5e-tls-certs") pod "model-serving-api-54d89bd5fc-vqssk" (UID: "f3cfb657-58c7-4501-aa5b-4883ee907a5e") : secret "model-serving-api-tls" not found Jul 27 21:39:29.773835 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.773806 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n52jd\" (UniqueName: \"kubernetes.io/projected/f3cfb657-58c7-4501-aa5b-4883ee907a5e-kube-api-access-n52jd\") pod \"model-serving-api-54d89bd5fc-vqssk\" (UID: \"f3cfb657-58c7-4501-aa5b-4883ee907a5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:29.875810 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.875724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:29.979233 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:29.979196 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:30.121149 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.121108 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:30.122041 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.122007 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts"] Jul 27 21:39:30.124305 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.124287 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.126983 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.126914 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 21:39:30.126983 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.126940 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 21:39:30.126983 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.126953 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 21:39:30.126983 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.126949 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 21:39:30.127345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.127327 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-p9txj\"" Jul 27 21:39:30.138315 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.138293 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts"] Jul 27 21:39:30.182790 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.182746 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:30.258464 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cfb657-58c7-4501-aa5b-4883ee907a5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-vqssk\" (UID: \"f3cfb657-58c7-4501-aa5b-4883ee907a5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:30.258644 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258480 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.258644 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjn2q\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-kube-api-access-xjn2q\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.258644 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258554 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.258850 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258734 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49b189f-fd89-4dcd-9688-a63739f441a4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.258850 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258776 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.258850 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258825 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.258991 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.258859 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.261464 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.261434 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3cfb657-58c7-4501-aa5b-4883ee907a5e-tls-certs\") pod \"model-serving-api-54d89bd5fc-vqssk\" (UID: \"f3cfb657-58c7-4501-aa5b-4883ee907a5e\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:30.359282 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.359247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49b189f-fd89-4dcd-9688-a63739f441a4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.359451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.359293 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.359451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.359323 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.359451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.359351 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.359451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.359399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.359451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.359421 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjn2q\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-kube-api-access-xjn2q\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.359757 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.359457 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.360436 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.360402 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.362116 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.362057 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.362993 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.362952 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49b189f-fd89-4dcd-9688-a63739f441a4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.363111 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.362994 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.363111 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.363013 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.370260 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.370229 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjn2q\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-kube-api-access-xjn2q\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.370391 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.370270 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nqrts\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.435843 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.435807 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:30.479457 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.479406 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:30.561910 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.561878 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:30.564461 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.564438 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ca527b1-69df-4373-94cc-e0a164209688-cert\") pod \"odh-model-controller-7998c67dc4-qhx8r\" (UID: \"5ca527b1-69df-4373-94cc-e0a164209688\") " pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:30.589468 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.589350 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts"] Jul 27 21:39:30.593468 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:30.593410 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49b189f_fd89_4dcd_9688_a63739f441a4.slice/crio-6800875949f565cc2b3a39a07fb9b4b3da7e53cdb8343cff1901ee180e397c02 WatchSource:0}: Error finding container 6800875949f565cc2b3a39a07fb9b4b3da7e53cdb8343cff1901ee180e397c02: Status 404 returned error can't find the container with id 6800875949f565cc2b3a39a07fb9b4b3da7e53cdb8343cff1901ee180e397c02 Jul 27 21:39:30.621275 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.621192 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vqssk"] Jul 27 21:39:30.623651 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:30.623624 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cfb657_58c7_4501_aa5b_4883ee907a5e.slice/crio-6df884037439290094e9111a781dadb982e1737aef80b81407bb86af261f7975 WatchSource:0}: Error finding container 6df884037439290094e9111a781dadb982e1737aef80b81407bb86af261f7975: Status 404 returned error can't find the container with id 6df884037439290094e9111a781dadb982e1737aef80b81407bb86af261f7975 Jul 27 21:39:30.624754 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.624730 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:30.771883 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:30.771851 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qhx8r"] Jul 27 21:39:30.774868 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:30.774838 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca527b1_69df_4373_94cc_e0a164209688.slice/crio-5f34fe805096f79ec04799903a8b140ebd9ca66d278cae9c408fd8870e58a810 WatchSource:0}: Error finding container 5f34fe805096f79ec04799903a8b140ebd9ca66d278cae9c408fd8870e58a810: Status 404 returned error can't find the container with id 5f34fe805096f79ec04799903a8b140ebd9ca66d278cae9c408fd8870e58a810 Jul 27 21:39:31.276616 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:31.276578 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:31.550371 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:31.550280 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" event={"ID":"d49b189f-fd89-4dcd-9688-a63739f441a4","Type":"ContainerStarted","Data":"6800875949f565cc2b3a39a07fb9b4b3da7e53cdb8343cff1901ee180e397c02"} Jul 27 21:39:31.551590 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:31.551561 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" event={"ID":"f3cfb657-58c7-4501-aa5b-4883ee907a5e","Type":"ContainerStarted","Data":"6df884037439290094e9111a781dadb982e1737aef80b81407bb86af261f7975"} Jul 27 21:39:31.552897 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:31.552871 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" event={"ID":"5ca527b1-69df-4373-94cc-e0a164209688","Type":"ContainerStarted","Data":"5f34fe805096f79ec04799903a8b140ebd9ca66d278cae9c408fd8870e58a810"} Jul 27 21:39:32.548180 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:32.548069 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:32.561318 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:32.561259 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" event={"ID":"4a92de9c-6da0-41be-908f-ba23b2de00ba","Type":"ContainerStarted","Data":"d1dcfbd0fb0d4401fa80b2091274716373d3edc4cccd8024110a46ef13f7d2b9"} Jul 27 21:39:32.580491 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:32.580421 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fd65f9cdf-8tcdx" podStartSLOduration=1.990976292 podStartE2EDuration="4.580397146s" totalImagesPullingTime="2.589420854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:32.578520572 +0000 UTC m=+340.986942604" watchObservedRunningTime="2026-07-27 21:39:32.580397146 +0000 UTC m=+340.988819177" Jul 27 21:39:35.308815 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.308775 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:39:35.309105 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.308882 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:39:35.577523 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.577425 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" event={"ID":"f3cfb657-58c7-4501-aa5b-4883ee907a5e","Type":"ContainerStarted","Data":"8932ac881fe490551a7b9f68e238b475fffbdced843c09b4890fc3d2d2fe7dd5"} Jul 27 21:39:35.577722 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.577563 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:39:35.579259 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.579230 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" event={"ID":"5ca527b1-69df-4373-94cc-e0a164209688","Type":"ContainerStarted","Data":"1084379a85d9186ae361b21019072aca9a81589bb3cbaf06dec95614b36a7e92"} Jul 27 21:39:35.579400 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.579298 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:35.581093 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.581067 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" event={"ID":"d49b189f-fd89-4dcd-9688-a63739f441a4","Type":"ContainerStarted","Data":"4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b"} Jul 27 21:39:35.581209 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.581197 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:35.611024 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.610954 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" podStartSLOduration=1.9382704670000002 podStartE2EDuration="6.610933762s" totalImagesPullingTime="4.672663295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:35.609234537 +0000 UTC m=+344.017656569" watchObservedRunningTime="2026-07-27 21:39:35.610933762 +0000 UTC m=+344.019355786" Jul 27 21:39:35.649384 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.649321 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" podStartSLOduration=0.936682052 podStartE2EDuration="5.649282241s" totalImagesPullingTime="4.712600189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:35.645945496 +0000 UTC m=+344.054367529" watchObservedRunningTime="2026-07-27 21:39:35.649282241 +0000 UTC m=+344.057704272" Jul 27 21:39:35.670849 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:35.670800 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" podStartSLOduration=3.135612226 podStartE2EDuration="7.670783979s" totalImagesPullingTime="4.535171753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:35.66972081 +0000 UTC m=+344.078142843" watchObservedRunningTime="2026-07-27 21:39:35.670783979 +0000 UTC m=+344.079206008" Jul 27 21:39:36.467796 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.467760 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:36.557657 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.557607 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:36.592137 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.592095 2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-nqrts container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 21:39:36.592326 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.592161 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" podUID="d49b189f-fd89-4dcd-9688-a63739f441a4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 21:39:36.790336 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.790254 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck"] Jul 27 21:39:36.794076 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.794051 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.797456 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.797435 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qhzrm\"" Jul 27 21:39:36.816914 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.816871 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:36.818776 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.818747 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck"] Jul 27 21:39:36.921585 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921546 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.921585 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921584 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.921840 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921612 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.921840 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921634 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.921840 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921736 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9c4l\" (UniqueName: \"kubernetes.io/projected/18a1ecc8-7323-4d14-84e6-e763daf75cb4-kube-api-access-q9c4l\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.921840 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.921840 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921794 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.922089 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921855 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:36.922089 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:36.921892 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023201 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023160 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023201 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023208 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023417 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023327 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023417 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023381 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023417 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023411 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023535 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023436 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9c4l\" (UniqueName: \"kubernetes.io/projected/18a1ecc8-7323-4d14-84e6-e763daf75cb4-kube-api-access-q9c4l\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023535 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023476 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023535 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023506 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023693 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023543 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023693 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023560 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023828 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023802 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023828 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023820 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.023930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.023875 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.024280 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.024258 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.025749 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.025727 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.026321 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.026306 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.049003 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.048935 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a1ecc8-7323-4d14-84e6-e763daf75cb4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.049476 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.049456 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9c4l\" (UniqueName: \"kubernetes.io/projected/18a1ecc8-7323-4d14-84e6-e763daf75cb4-kube-api-access-q9c4l\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c66lck\" (UID: \"18a1ecc8-7323-4d14-84e6-e763daf75cb4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.098026 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.097983 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:37.104285 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.104254 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:37.194318 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.194272 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:37.246158 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.246112 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck"] Jul 27 21:39:37.249027 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:37.249002 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a1ecc8_7323_4d14_84e6_e763daf75cb4.slice/crio-c289451d4b7d2231b992e3123cf4e1a811d3d3bdc75c28f5d41e34c647c577a6 WatchSource:0}: Error finding container c289451d4b7d2231b992e3123cf4e1a811d3d3bdc75c28f5d41e34c647c577a6: Status 404 returned error can't find the container with id c289451d4b7d2231b992e3123cf4e1a811d3d3bdc75c28f5d41e34c647c577a6 Jul 27 21:39:37.592690 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:37.592634 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" event={"ID":"18a1ecc8-7323-4d14-84e6-e763daf75cb4","Type":"ContainerStarted","Data":"c289451d4b7d2231b992e3123cf4e1a811d3d3bdc75c28f5d41e34c647c577a6"} Jul 27 21:39:39.052112 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.052078 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:39.170486 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.170454 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6"] Jul 27 21:39:39.177687 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.177637 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh"] Jul 27 21:39:39.177858 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.177766 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.181990 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.181089 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.184930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.184625 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 21:39:39.184930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.184637 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nl4nq\"" Jul 27 21:39:39.184930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.184723 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-54sr7\"" Jul 27 21:39:39.184930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.184635 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 21:39:39.201700 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.201614 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6"] Jul 27 21:39:39.203736 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.203707 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh"] Jul 27 21:39:39.204493 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.204270 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:39.344628 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.344522 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b878611-46b4-4b23-a882-11dcbc016cfa-cert\") pod \"kserve-controller-manager-d6f4b65b8-w8mt6\" (UID: \"3b878611-46b4-4b23-a882-11dcbc016cfa\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.344628 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.344578 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e6e129f-313e-499a-80c5-a623d02b30f0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wqkhh\" (UID: \"2e6e129f-313e-499a-80c5-a623d02b30f0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.344628 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.344605 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4qh\" (UniqueName: \"kubernetes.io/projected/2e6e129f-313e-499a-80c5-a623d02b30f0-kube-api-access-rk4qh\") pod \"llmisvc-controller-manager-777c6bd7d5-wqkhh\" (UID: \"2e6e129f-313e-499a-80c5-a623d02b30f0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.344933 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.344723 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jcb\" (UniqueName: \"kubernetes.io/projected/3b878611-46b4-4b23-a882-11dcbc016cfa-kube-api-access-s5jcb\") pod \"kserve-controller-manager-d6f4b65b8-w8mt6\" (UID: \"3b878611-46b4-4b23-a882-11dcbc016cfa\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.396618 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.396580 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:39.445489 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.445453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b878611-46b4-4b23-a882-11dcbc016cfa-cert\") pod \"kserve-controller-manager-d6f4b65b8-w8mt6\" (UID: \"3b878611-46b4-4b23-a882-11dcbc016cfa\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.445665 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.445495 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e6e129f-313e-499a-80c5-a623d02b30f0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wqkhh\" (UID: \"2e6e129f-313e-499a-80c5-a623d02b30f0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.445665 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.445528 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4qh\" (UniqueName: \"kubernetes.io/projected/2e6e129f-313e-499a-80c5-a623d02b30f0-kube-api-access-rk4qh\") pod \"llmisvc-controller-manager-777c6bd7d5-wqkhh\" (UID: \"2e6e129f-313e-499a-80c5-a623d02b30f0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.445665 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.445582 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jcb\" (UniqueName: \"kubernetes.io/projected/3b878611-46b4-4b23-a882-11dcbc016cfa-kube-api-access-s5jcb\") pod \"kserve-controller-manager-d6f4b65b8-w8mt6\" (UID: \"3b878611-46b4-4b23-a882-11dcbc016cfa\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.448974 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.448943 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b878611-46b4-4b23-a882-11dcbc016cfa-cert\") pod \"kserve-controller-manager-d6f4b65b8-w8mt6\" (UID: \"3b878611-46b4-4b23-a882-11dcbc016cfa\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.449114 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.448943 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e6e129f-313e-499a-80c5-a623d02b30f0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wqkhh\" (UID: \"2e6e129f-313e-499a-80c5-a623d02b30f0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.466181 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.466151 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4qh\" (UniqueName: \"kubernetes.io/projected/2e6e129f-313e-499a-80c5-a623d02b30f0-kube-api-access-rk4qh\") pod \"llmisvc-controller-manager-777c6bd7d5-wqkhh\" (UID: \"2e6e129f-313e-499a-80c5-a623d02b30f0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.468437 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.468404 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jcb\" (UniqueName: \"kubernetes.io/projected/3b878611-46b4-4b23-a882-11dcbc016cfa-kube-api-access-s5jcb\") pod \"kserve-controller-manager-d6f4b65b8-w8mt6\" (UID: \"3b878611-46b4-4b23-a882-11dcbc016cfa\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.493305 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.493259 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:39.502698 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.502651 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:39.563086 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.563033 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:39.590226 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.590200 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:39.680052 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.679925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:39.788073 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:39.788033 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:39:40.083495 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.083446 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:39:40.083867 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.083541 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:39:40.083867 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.083570 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:39:40.160407 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.160375 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh"] Jul 27 21:39:40.162068 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:40.162021 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e6e129f_313e_499a_80c5_a623d02b30f0.slice/crio-4f4e3cf366dc60065bbdc05ec3696e52b7e225ccf0d10e643482905508df75e4 WatchSource:0}: Error finding container 4f4e3cf366dc60065bbdc05ec3696e52b7e225ccf0d10e643482905508df75e4: Status 404 returned error can't find the container with id 4f4e3cf366dc60065bbdc05ec3696e52b7e225ccf0d10e643482905508df75e4 Jul 27 21:39:40.180612 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.180587 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6"] Jul 27 21:39:40.183043 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:39:40.183015 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b878611_46b4_4b23_a882_11dcbc016cfa.slice/crio-39d1acf33df4245ee3f6df39ef2482f8f28379d6b58c7b415ef6ca07fbaab082 WatchSource:0}: Error finding container 39d1acf33df4245ee3f6df39ef2482f8f28379d6b58c7b415ef6ca07fbaab082: Status 404 returned error can't find the container with id 39d1acf33df4245ee3f6df39ef2482f8f28379d6b58c7b415ef6ca07fbaab082 Jul 27 21:39:40.342026 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.341939 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts"] Jul 27 21:39:40.342204 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.342174 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" podUID="d49b189f-fd89-4dcd-9688-a63739f441a4" containerName="discovery" containerID="cri-o://4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b" gracePeriod=30 Jul 27 21:39:40.587932 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.587908 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:40.609739 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.609633 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" event={"ID":"2e6e129f-313e-499a-80c5-a623d02b30f0","Type":"ContainerStarted","Data":"4f4e3cf366dc60065bbdc05ec3696e52b7e225ccf0d10e643482905508df75e4"} Jul 27 21:39:40.611171 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.611140 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" event={"ID":"3b878611-46b4-4b23-a882-11dcbc016cfa","Type":"ContainerStarted","Data":"39d1acf33df4245ee3f6df39ef2482f8f28379d6b58c7b415ef6ca07fbaab082"} Jul 27 21:39:40.612561 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.612537 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" event={"ID":"18a1ecc8-7323-4d14-84e6-e763daf75cb4","Type":"ContainerStarted","Data":"0fa2b54e986fc566ddabd5adbe14f81a1e43ded34f7bf3e2b0329e03268a06a2"} Jul 27 21:39:40.613897 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.613873 2556 generic.go:356] "Generic (PLEG): container finished" podID="d49b189f-fd89-4dcd-9688-a63739f441a4" containerID="4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b" exitCode=0 Jul 27 21:39:40.613993 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.613906 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" event={"ID":"d49b189f-fd89-4dcd-9688-a63739f441a4","Type":"ContainerDied","Data":"4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b"} Jul 27 21:39:40.613993 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.613915 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" Jul 27 21:39:40.613993 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.613931 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts" event={"ID":"d49b189f-fd89-4dcd-9688-a63739f441a4","Type":"ContainerDied","Data":"6800875949f565cc2b3a39a07fb9b4b3da7e53cdb8343cff1901ee180e397c02"} Jul 27 21:39:40.613993 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.613956 2556 scope.go:117] "RemoveContainer" containerID="4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b" Jul 27 21:39:40.622004 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.621982 2556 scope.go:117] "RemoveContainer" containerID="4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b" Jul 27 21:39:40.622291 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:39:40.622271 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b\": container with ID starting with 4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b not found: ID does not exist" containerID="4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b" Jul 27 21:39:40.622349 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.622301 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b"} err="failed to get container status \"4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b\": rpc error: code = NotFound desc = could not find container \"4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b\": container with ID starting with 4d00815b7d1a43c84389dce75b8280732d4ac9914f46579b04705e434b873e6b not found: ID does not exist" Jul 27 21:39:40.756447 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.756414 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-dns-cert\") pod \"d49b189f-fd89-4dcd-9688-a63739f441a4\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " Jul 27 21:39:40.756597 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.756469 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49b189f-fd89-4dcd-9688-a63739f441a4-local-certs\") pod \"d49b189f-fd89-4dcd-9688-a63739f441a4\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " Jul 27 21:39:40.756597 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.756504 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-cacerts\") pod \"d49b189f-fd89-4dcd-9688-a63739f441a4\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " Jul 27 21:39:40.756597 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.756528 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-token\") pod \"d49b189f-fd89-4dcd-9688-a63739f441a4\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " Jul 27 21:39:40.756597 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.756574 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjn2q\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-kube-api-access-xjn2q\") pod \"d49b189f-fd89-4dcd-9688-a63739f441a4\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " Jul 27 21:39:40.756805 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.756614 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-ca-configmap\") pod \"d49b189f-fd89-4dcd-9688-a63739f441a4\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " Jul 27 21:39:40.756805 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.756641 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-kubeconfig\") pod \"d49b189f-fd89-4dcd-9688-a63739f441a4\" (UID: \"d49b189f-fd89-4dcd-9688-a63739f441a4\") " Jul 27 21:39:40.757724 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.757066 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d49b189f-fd89-4dcd-9688-a63739f441a4" (UID: "d49b189f-fd89-4dcd-9688-a63739f441a4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 21:39:40.759115 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.759089 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d49b189f-fd89-4dcd-9688-a63739f441a4" (UID: "d49b189f-fd89-4dcd-9688-a63739f441a4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:39:40.759228 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.759209 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49b189f-fd89-4dcd-9688-a63739f441a4-local-certs" (OuterVolumeSpecName: "local-certs") pod "d49b189f-fd89-4dcd-9688-a63739f441a4" (UID: "d49b189f-fd89-4dcd-9688-a63739f441a4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:39:40.759228 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.759211 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d49b189f-fd89-4dcd-9688-a63739f441a4" (UID: "d49b189f-fd89-4dcd-9688-a63739f441a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:39:40.759374 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.759358 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-cacerts" (OuterVolumeSpecName: "cacerts") pod "d49b189f-fd89-4dcd-9688-a63739f441a4" (UID: "d49b189f-fd89-4dcd-9688-a63739f441a4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:39:40.759654 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.759636 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d49b189f-fd89-4dcd-9688-a63739f441a4" (UID: "d49b189f-fd89-4dcd-9688-a63739f441a4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:39:40.759771 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.759748 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-kube-api-access-xjn2q" (OuterVolumeSpecName: "kube-api-access-xjn2q") pod "d49b189f-fd89-4dcd-9688-a63739f441a4" (UID: "d49b189f-fd89-4dcd-9688-a63739f441a4"). InnerVolumeSpecName "kube-api-access-xjn2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:39:40.857854 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.857814 2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-dns-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:39:40.857854 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.857853 2556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49b189f-fd89-4dcd-9688-a63739f441a4-local-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:39:40.857854 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.857868 2556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-cacerts\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:39:40.858149 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.857880 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:39:40.858149 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.857894 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjn2q\" (UniqueName: \"kubernetes.io/projected/d49b189f-fd89-4dcd-9688-a63739f441a4-kube-api-access-xjn2q\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:39:40.858149 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.857908 2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-csr-ca-configmap\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:39:40.858149 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.857921 2556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d49b189f-fd89-4dcd-9688-a63739f441a4-istio-kubeconfig\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:39:40.947725 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.947646 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" podStartSLOduration=2.115639173 podStartE2EDuration="4.947622753s" totalImagesPullingTime="2.83198358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:40.668314138 +0000 UTC m=+349.076736168" watchObservedRunningTime="2026-07-27 21:39:40.947622753 +0000 UTC m=+349.356044785" Jul 27 21:39:40.948245 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.948207 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts"] Jul 27 21:39:40.953238 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:40.953181 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nqrts"] Jul 27 21:39:41.105323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:41.105193 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:41.111339 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:41.111311 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:41.620015 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:41.619987 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:41.620868 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:41.620845 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c66lck" Jul 27 21:39:42.193485 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:42.193384 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49b189f-fd89-4dcd-9688-a63739f441a4" path="/var/lib/kubelet/pods/d49b189f-fd89-4dcd-9688-a63739f441a4/volumes" Jul 27 21:39:43.637170 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:43.637132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" event={"ID":"3b878611-46b4-4b23-a882-11dcbc016cfa","Type":"ContainerStarted","Data":"1d066fec10af9afcdc925ffb28c4aa76fd61117269074fe99a7be5d9bfdf94e0"} Jul 27 21:39:43.637629 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:43.637243 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:39:43.638758 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:43.638734 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" event={"ID":"2e6e129f-313e-499a-80c5-a623d02b30f0","Type":"ContainerStarted","Data":"d2b5afb6c0c192aedb6f6e3ad323be1d360d3f095c1d545e114ad09dd4f325dd"} Jul 27 21:39:43.639005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:43.638992 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:39:43.654778 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:43.654728 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" podStartSLOduration=1.320320855 podStartE2EDuration="4.654713071s" totalImagesPullingTime="3.334392216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:43.653023141 +0000 UTC m=+352.061445171" watchObservedRunningTime="2026-07-27 21:39:43.654713071 +0000 UTC m=+352.063135100" Jul 27 21:39:43.668948 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:43.668898 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" podStartSLOduration=1.311375918 podStartE2EDuration="4.668881906s" totalImagesPullingTime="3.357505988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:43.668196714 +0000 UTC m=+352.076618744" watchObservedRunningTime="2026-07-27 21:39:43.668881906 +0000 UTC m=+352.077303936" Jul 27 21:39:46.590892 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:46.590861 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qhx8r" Jul 27 21:39:46.592631 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:39:46.592614 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vqssk" Jul 27 21:40:14.645291 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:14.645257 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wqkhh" Jul 27 21:40:14.648206 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:14.648182 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8mt6" Jul 27 21:40:26.966692 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:26.966634 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:27.072224 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.072190 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:27.111761 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.111727 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mw8gj"] Jul 27 21:40:27.112206 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.112183 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49b189f-fd89-4dcd-9688-a63739f441a4" containerName="discovery" Jul 27 21:40:27.112206 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.112205 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49b189f-fd89-4dcd-9688-a63739f441a4" containerName="discovery" Jul 27 21:40:27.112411 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.112327 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49b189f-fd89-4dcd-9688-a63739f441a4" containerName="discovery" Jul 27 21:40:27.118880 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.118854 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" Jul 27 21:40:27.121911 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.121889 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6mjdg\"" Jul 27 21:40:27.122042 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.121936 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 21:40:27.122794 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.122769 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 21:40:27.130529 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.130504 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mw8gj"] Jul 27 21:40:27.190528 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.190498 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:27.242840 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.242752 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj79b\" (UniqueName: \"kubernetes.io/projected/67b5f27e-f69b-47d0-a3db-280486c182c2-kube-api-access-tj79b\") pod \"kuadrant-operator-catalog-mw8gj\" (UID: \"67b5f27e-f69b-47d0-a3db-280486c182c2\") " pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" Jul 27 21:40:27.344176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.344137 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj79b\" (UniqueName: \"kubernetes.io/projected/67b5f27e-f69b-47d0-a3db-280486c182c2-kube-api-access-tj79b\") pod \"kuadrant-operator-catalog-mw8gj\" (UID: \"67b5f27e-f69b-47d0-a3db-280486c182c2\") " pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" Jul 27 21:40:27.353784 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.353763 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj79b\" (UniqueName: \"kubernetes.io/projected/67b5f27e-f69b-47d0-a3db-280486c182c2-kube-api-access-tj79b\") pod \"kuadrant-operator-catalog-mw8gj\" (UID: \"67b5f27e-f69b-47d0-a3db-280486c182c2\") " pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" Jul 27 21:40:27.429595 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.429565 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" Jul 27 21:40:27.453331 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.453299 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mw8gj"] Jul 27 21:40:27.560314 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.560291 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mw8gj"] Jul 27 21:40:27.562981 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:27.562952 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b5f27e_f69b_47d0_a3db_280486c182c2.slice/crio-1cb81b991da83f33203547db1db2bc987a1e0a1c8c3bbd1eaf376f94102dabbf WatchSource:0}: Error finding container 1cb81b991da83f33203547db1db2bc987a1e0a1c8c3bbd1eaf376f94102dabbf: Status 404 returned error can't find the container with id 1cb81b991da83f33203547db1db2bc987a1e0a1c8c3bbd1eaf376f94102dabbf Jul 27 21:40:27.663644 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.663604 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4x4"] Jul 27 21:40:27.666509 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.666485 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:27.675282 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.675258 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4x4"] Jul 27 21:40:27.747856 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.747816 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfns\" (UniqueName: \"kubernetes.io/projected/f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f-kube-api-access-ggfns\") pod \"kuadrant-operator-catalog-wl4x4\" (UID: \"f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f\") " pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:27.843873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.843770 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" event={"ID":"67b5f27e-f69b-47d0-a3db-280486c182c2","Type":"ContainerStarted","Data":"1cb81b991da83f33203547db1db2bc987a1e0a1c8c3bbd1eaf376f94102dabbf"} Jul 27 21:40:27.848713 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.848660 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfns\" (UniqueName: \"kubernetes.io/projected/f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f-kube-api-access-ggfns\") pod \"kuadrant-operator-catalog-wl4x4\" (UID: \"f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f\") " pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:27.859652 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.859620 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfns\" (UniqueName: \"kubernetes.io/projected/f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f-kube-api-access-ggfns\") pod \"kuadrant-operator-catalog-wl4x4\" (UID: \"f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f\") " pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:27.895225 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.895192 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:27.975808 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:27.975774 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:28.109040 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:28.108947 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:28.135478 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:28.135449 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4x4"] Jul 27 21:40:28.145362 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:28.145332 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ad1d69_8928_4d7f_bdc4_b9ed2001e19f.slice/crio-99bb86f19ebd13323d53e97d113a8fc3add9442c4d749fe753955e505a11b929 WatchSource:0}: Error finding container 99bb86f19ebd13323d53e97d113a8fc3add9442c4d749fe753955e505a11b929: Status 404 returned error can't find the container with id 99bb86f19ebd13323d53e97d113a8fc3add9442c4d749fe753955e505a11b929 Jul 27 21:40:28.852240 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:28.852199 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" event={"ID":"f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f","Type":"ContainerStarted","Data":"99bb86f19ebd13323d53e97d113a8fc3add9442c4d749fe753955e505a11b929"} Jul 27 21:40:29.858448 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:29.858404 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" event={"ID":"67b5f27e-f69b-47d0-a3db-280486c182c2","Type":"ContainerStarted","Data":"5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34"} Jul 27 21:40:29.858448 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:29.858431 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" podUID="67b5f27e-f69b-47d0-a3db-280486c182c2" containerName="registry-server" containerID="cri-o://5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34" gracePeriod=2 Jul 27 21:40:29.860178 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:29.860151 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" event={"ID":"f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f","Type":"ContainerStarted","Data":"0d2cc28d0e4f8fe5b6be1ebdc95b591743a51f6d12c5492b65872f91b628e98d"} Jul 27 21:40:29.883823 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:29.883782 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" podStartSLOduration=0.772094721 podStartE2EDuration="2.88376693s" totalImagesPullingTime="2.111672209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:29.88253245 +0000 UTC m=+398.290954480" watchObservedRunningTime="2026-07-27 21:40:29.88376693 +0000 UTC m=+398.292188961" Jul 27 21:40:29.899626 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:29.899579 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" podStartSLOduration=1.367885175 podStartE2EDuration="2.89956311s" totalImagesPullingTime="1.531677935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:29.898811694 +0000 UTC m=+398.307233725" watchObservedRunningTime="2026-07-27 21:40:29.89956311 +0000 UTC m=+398.307985139" Jul 27 21:40:30.098857 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.098834 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" Jul 27 21:40:30.171903 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.171875 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj79b\" (UniqueName: \"kubernetes.io/projected/67b5f27e-f69b-47d0-a3db-280486c182c2-kube-api-access-tj79b\") pod \"67b5f27e-f69b-47d0-a3db-280486c182c2\" (UID: \"67b5f27e-f69b-47d0-a3db-280486c182c2\") " Jul 27 21:40:30.174144 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.174112 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b5f27e-f69b-47d0-a3db-280486c182c2-kube-api-access-tj79b" (OuterVolumeSpecName: "kube-api-access-tj79b") pod "67b5f27e-f69b-47d0-a3db-280486c182c2" (UID: "67b5f27e-f69b-47d0-a3db-280486c182c2"). InnerVolumeSpecName "kube-api-access-tj79b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:40:30.272896 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.272816 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj79b\" (UniqueName: \"kubernetes.io/projected/67b5f27e-f69b-47d0-a3db-280486c182c2-kube-api-access-tj79b\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:30.866588 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.866555 2556 generic.go:356] "Generic (PLEG): container finished" podID="67b5f27e-f69b-47d0-a3db-280486c182c2" containerID="5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34" exitCode=0 Jul 27 21:40:30.867128 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.866627 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" Jul 27 21:40:30.867128 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.866636 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" event={"ID":"67b5f27e-f69b-47d0-a3db-280486c182c2","Type":"ContainerDied","Data":"5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34"} Jul 27 21:40:30.867128 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.866708 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mw8gj" event={"ID":"67b5f27e-f69b-47d0-a3db-280486c182c2","Type":"ContainerDied","Data":"1cb81b991da83f33203547db1db2bc987a1e0a1c8c3bbd1eaf376f94102dabbf"} Jul 27 21:40:30.867128 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.866730 2556 scope.go:117] "RemoveContainer" containerID="5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34" Jul 27 21:40:30.875842 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.875824 2556 scope.go:117] "RemoveContainer" containerID="5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34" Jul 27 21:40:30.876142 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:40:30.876124 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34\": container with ID starting with 5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34 not found: ID does not exist" containerID="5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34" Jul 27 21:40:30.876193 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.876152 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34"} err="failed to get container status \"5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34\": rpc error: code = NotFound desc = could not find container \"5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34\": container with ID starting with 5709a9c2d581430488354598ebc38329c240f97c15cd446635afda4ff549ee34 not found: ID does not exist" Jul 27 21:40:30.893187 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.893158 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mw8gj"] Jul 27 21:40:30.896159 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:30.896136 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mw8gj"] Jul 27 21:40:32.195786 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:32.195751 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b5f27e-f69b-47d0-a3db-280486c182c2" path="/var/lib/kubelet/pods/67b5f27e-f69b-47d0-a3db-280486c182c2/volumes" Jul 27 21:40:36.742264 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:36.742230 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:36.814521 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:36.814489 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:37.976170 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:37.976128 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:37.976607 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:37.976192 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:37.998092 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:37.998067 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:38.923065 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:38.923036 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wl4x4" Jul 27 21:40:42.713126 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.713087 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8"] Jul 27 21:40:42.713481 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.713381 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b5f27e-f69b-47d0-a3db-280486c182c2" containerName="registry-server" Jul 27 21:40:42.713481 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.713396 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b5f27e-f69b-47d0-a3db-280486c182c2" containerName="registry-server" Jul 27 21:40:42.713481 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.713454 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b5f27e-f69b-47d0-a3db-280486c182c2" containerName="registry-server" Jul 27 21:40:42.717938 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.717917 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.720645 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.720622 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vx4tb\"" Jul 27 21:40:42.727391 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.727359 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8"] Jul 27 21:40:42.877045 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.877008 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.877260 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.877071 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2g6\" (UniqueName: \"kubernetes.io/projected/991508b7-4c79-4b6b-98eb-aa26853b3758-kube-api-access-8q2g6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.877260 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.877166 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.978349 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.978254 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.978349 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.978325 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.978558 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.978403 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2g6\" (UniqueName: \"kubernetes.io/projected/991508b7-4c79-4b6b-98eb-aa26853b3758-kube-api-access-8q2g6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.978751 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.978730 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.978804 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.978744 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:42.987468 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:42.987441 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2g6\" (UniqueName: \"kubernetes.io/projected/991508b7-4c79-4b6b-98eb-aa26853b3758-kube-api-access-8q2g6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:43.029116 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.029081 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:43.115686 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.115644 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4"] Jul 27 21:40:43.119207 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.119183 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.130923 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.130892 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4"] Jul 27 21:40:43.174577 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.174553 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8"] Jul 27 21:40:43.176708 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:43.176684 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991508b7_4c79_4b6b_98eb_aa26853b3758.slice/crio-957627eb46d1e6c4dd6e53deb23042b30bb3ddc80a843d6443d3ede1dd06ef5e WatchSource:0}: Error finding container 957627eb46d1e6c4dd6e53deb23042b30bb3ddc80a843d6443d3ede1dd06ef5e: Status 404 returned error can't find the container with id 957627eb46d1e6c4dd6e53deb23042b30bb3ddc80a843d6443d3ede1dd06ef5e Jul 27 21:40:43.284941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.284903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.285112 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.284960 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdn79\" (UniqueName: \"kubernetes.io/projected/b0004e05-a625-4091-bcdc-e21f0646f7e3-kube-api-access-qdn79\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.285112 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.284994 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.386393 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.386356 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.386393 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.386399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdn79\" (UniqueName: \"kubernetes.io/projected/b0004e05-a625-4091-bcdc-e21f0646f7e3-kube-api-access-qdn79\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.386605 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.386423 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.386767 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.386751 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.386843 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.386821 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.397554 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.397528 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdn79\" (UniqueName: \"kubernetes.io/projected/b0004e05-a625-4091-bcdc-e21f0646f7e3-kube-api-access-qdn79\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.431047 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.431001 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:43.509793 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.509701 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk"] Jul 27 21:40:43.513309 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.513285 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.523649 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.523372 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk"] Jul 27 21:40:43.573076 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.573036 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4"] Jul 27 21:40:43.575222 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:43.575189 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0004e05_a625_4091_bcdc_e21f0646f7e3.slice/crio-febe5f27e0616ab089eef5c31ae2dace25a218975ad89e9086e07b4fa7632823 WatchSource:0}: Error finding container febe5f27e0616ab089eef5c31ae2dace25a218975ad89e9086e07b4fa7632823: Status 404 returned error can't find the container with id febe5f27e0616ab089eef5c31ae2dace25a218975ad89e9086e07b4fa7632823 Jul 27 21:40:43.689917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.689886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.689917 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.689921 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4flbs\" (UniqueName: \"kubernetes.io/projected/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-kube-api-access-4flbs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.690101 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.689947 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.790485 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.790403 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.790485 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.790432 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4flbs\" (UniqueName: \"kubernetes.io/projected/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-kube-api-access-4flbs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.790485 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.790456 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.790977 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.790775 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.790977 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.790857 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.799404 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.799381 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4flbs\" (UniqueName: \"kubernetes.io/projected/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-kube-api-access-4flbs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.826420 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.826386 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:43.919696 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.919645 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr"] Jul 27 21:40:43.923762 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.923736 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:43.925822 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.925785 2556 generic.go:356] "Generic (PLEG): container finished" podID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerID="55e62cf6db8e0ddf98ccab8723f2becb39fe65bc5f51ce1100f919bca004460d" exitCode=0 Jul 27 21:40:43.925961 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.925871 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" event={"ID":"b0004e05-a625-4091-bcdc-e21f0646f7e3","Type":"ContainerDied","Data":"55e62cf6db8e0ddf98ccab8723f2becb39fe65bc5f51ce1100f919bca004460d"} Jul 27 21:40:43.925961 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.925922 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" event={"ID":"b0004e05-a625-4091-bcdc-e21f0646f7e3","Type":"ContainerStarted","Data":"febe5f27e0616ab089eef5c31ae2dace25a218975ad89e9086e07b4fa7632823"} Jul 27 21:40:43.927712 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.927662 2556 generic.go:356] "Generic (PLEG): container finished" podID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerID="8b8725a151a22e2a3e60171136bac9ff8ca9a336ddf95f37f649fdde138e9a88" exitCode=0 Jul 27 21:40:43.927831 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.927704 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" event={"ID":"991508b7-4c79-4b6b-98eb-aa26853b3758","Type":"ContainerDied","Data":"8b8725a151a22e2a3e60171136bac9ff8ca9a336ddf95f37f649fdde138e9a88"} Jul 27 21:40:43.927831 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.927752 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" event={"ID":"991508b7-4c79-4b6b-98eb-aa26853b3758","Type":"ContainerStarted","Data":"957627eb46d1e6c4dd6e53deb23042b30bb3ddc80a843d6443d3ede1dd06ef5e"} Jul 27 21:40:43.937743 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.937719 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr"] Jul 27 21:40:43.970542 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:43.970475 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk"] Jul 27 21:40:43.972689 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:43.972649 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d37d026_6f4b_407e_9f9c_a65fb2fa777b.slice/crio-2de99a5535fad0411784e13293abef788c70f4ff5ffdf0afd7bac828edad131e WatchSource:0}: Error finding container 2de99a5535fad0411784e13293abef788c70f4ff5ffdf0afd7bac828edad131e: Status 404 returned error can't find the container with id 2de99a5535fad0411784e13293abef788c70f4ff5ffdf0afd7bac828edad131e Jul 27 21:40:44.093368 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.093341 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6szz\" (UniqueName: \"kubernetes.io/projected/3290fdcd-a808-48ba-8d1f-be7e851e9546-kube-api-access-s6szz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.093496 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.093461 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.093563 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.093538 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.194734 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.194703 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.194921 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.194751 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.194921 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.194790 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6szz\" (UniqueName: \"kubernetes.io/projected/3290fdcd-a808-48ba-8d1f-be7e851e9546-kube-api-access-s6szz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.195124 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.195103 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.195182 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.195123 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.204123 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.204091 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6szz\" (UniqueName: \"kubernetes.io/projected/3290fdcd-a808-48ba-8d1f-be7e851e9546-kube-api-access-s6szz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.236864 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.236824 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:44.370263 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.370239 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr"] Jul 27 21:40:44.372106 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:44.372084 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3290fdcd_a808_48ba_8d1f_be7e851e9546.slice/crio-eae2bae18e34a6d7f4ca03401bfb2d0b8d88da496d482ca5d4654e37ff6144ef WatchSource:0}: Error finding container eae2bae18e34a6d7f4ca03401bfb2d0b8d88da496d482ca5d4654e37ff6144ef: Status 404 returned error can't find the container with id eae2bae18e34a6d7f4ca03401bfb2d0b8d88da496d482ca5d4654e37ff6144ef Jul 27 21:40:44.933526 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.933446 2556 generic.go:356] "Generic (PLEG): container finished" podID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerID="06afb9363a30cc996ad0d0e2eb1861cd6331e00de4d6bdb9b0df66be70b0c671" exitCode=0 Jul 27 21:40:44.933963 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.933536 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" event={"ID":"4d37d026-6f4b-407e-9f9c-a65fb2fa777b","Type":"ContainerDied","Data":"06afb9363a30cc996ad0d0e2eb1861cd6331e00de4d6bdb9b0df66be70b0c671"} Jul 27 21:40:44.933963 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.933598 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" event={"ID":"4d37d026-6f4b-407e-9f9c-a65fb2fa777b","Type":"ContainerStarted","Data":"2de99a5535fad0411784e13293abef788c70f4ff5ffdf0afd7bac828edad131e"} Jul 27 21:40:44.935497 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.935468 2556 generic.go:356] "Generic (PLEG): container finished" podID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerID="171b7454f5cc4dd43c7f30038ab6edbd3a6e2de5a3fd7f05c2107a3ee43ded2d" exitCode=0 Jul 27 21:40:44.935609 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.935546 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" event={"ID":"b0004e05-a625-4091-bcdc-e21f0646f7e3","Type":"ContainerDied","Data":"171b7454f5cc4dd43c7f30038ab6edbd3a6e2de5a3fd7f05c2107a3ee43ded2d"} Jul 27 21:40:44.937570 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.937551 2556 generic.go:356] "Generic (PLEG): container finished" podID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerID="a7e7d13634b96d14a9be6b17ecf6f594da66c482af6a160f2853d55cbc993f99" exitCode=0 Jul 27 21:40:44.937686 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.937632 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" event={"ID":"991508b7-4c79-4b6b-98eb-aa26853b3758","Type":"ContainerDied","Data":"a7e7d13634b96d14a9be6b17ecf6f594da66c482af6a160f2853d55cbc993f99"} Jul 27 21:40:44.939320 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.939270 2556 generic.go:356] "Generic (PLEG): container finished" podID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerID="7d7a7bdc6186299dcdb4be496ad7e4dc630e09ad1a1fa4d17d663db0bb1837d3" exitCode=0 Jul 27 21:40:44.939425 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.939374 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" event={"ID":"3290fdcd-a808-48ba-8d1f-be7e851e9546","Type":"ContainerDied","Data":"7d7a7bdc6186299dcdb4be496ad7e4dc630e09ad1a1fa4d17d663db0bb1837d3"} Jul 27 21:40:44.939425 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:44.939404 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" event={"ID":"3290fdcd-a808-48ba-8d1f-be7e851e9546","Type":"ContainerStarted","Data":"eae2bae18e34a6d7f4ca03401bfb2d0b8d88da496d482ca5d4654e37ff6144ef"} Jul 27 21:40:45.947548 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.947516 2556 generic.go:356] "Generic (PLEG): container finished" podID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerID="135cdbb68029e9d9285d1deba78e0c7527819c5947b2c7eaa70359de8d8717cd" exitCode=0 Jul 27 21:40:45.947922 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.947568 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" event={"ID":"4d37d026-6f4b-407e-9f9c-a65fb2fa777b","Type":"ContainerDied","Data":"135cdbb68029e9d9285d1deba78e0c7527819c5947b2c7eaa70359de8d8717cd"} Jul 27 21:40:45.949916 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.949891 2556 generic.go:356] "Generic (PLEG): container finished" podID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerID="9439ae625f231a6bc9cbb042caafaf060d6d9f8324c57efd1c0a4718ea23d50f" exitCode=0 Jul 27 21:40:45.950002 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.949971 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" event={"ID":"b0004e05-a625-4091-bcdc-e21f0646f7e3","Type":"ContainerDied","Data":"9439ae625f231a6bc9cbb042caafaf060d6d9f8324c57efd1c0a4718ea23d50f"} Jul 27 21:40:45.951941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.951923 2556 generic.go:356] "Generic (PLEG): container finished" podID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerID="ac182713f519f2d43c12b43b2e273327b631f39fb00376b54abc51812b5a3790" exitCode=0 Jul 27 21:40:45.952033 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.951984 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" event={"ID":"991508b7-4c79-4b6b-98eb-aa26853b3758","Type":"ContainerDied","Data":"ac182713f519f2d43c12b43b2e273327b631f39fb00376b54abc51812b5a3790"} Jul 27 21:40:45.953979 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.953864 2556 generic.go:356] "Generic (PLEG): container finished" podID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerID="4bcc11bc67c8abb4d7a7b597e0196c609399047ad10a58be57584bcd9383b8cb" exitCode=0 Jul 27 21:40:45.953979 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:45.953942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" event={"ID":"3290fdcd-a808-48ba-8d1f-be7e851e9546","Type":"ContainerDied","Data":"4bcc11bc67c8abb4d7a7b597e0196c609399047ad10a58be57584bcd9383b8cb"} Jul 27 21:40:46.960301 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:46.960266 2556 generic.go:356] "Generic (PLEG): container finished" podID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerID="caf60bded986f191bc18310c477aba560c179d258a3d6510aa2d4e5f74268c48" exitCode=0 Jul 27 21:40:46.960747 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:46.960353 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" event={"ID":"3290fdcd-a808-48ba-8d1f-be7e851e9546","Type":"ContainerDied","Data":"caf60bded986f191bc18310c477aba560c179d258a3d6510aa2d4e5f74268c48"} Jul 27 21:40:46.962490 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:46.962470 2556 generic.go:356] "Generic (PLEG): container finished" podID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerID="63d76bb86a2213f150d45429d59a963aabb785d0386a3ce83947a0def5e94933" exitCode=0 Jul 27 21:40:46.962612 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:46.962560 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" event={"ID":"4d37d026-6f4b-407e-9f9c-a65fb2fa777b","Type":"ContainerDied","Data":"63d76bb86a2213f150d45429d59a963aabb785d0386a3ce83947a0def5e94933"} Jul 27 21:40:47.095472 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.095447 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:47.123463 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.123435 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:47.220386 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.220308 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-bundle\") pod \"991508b7-4c79-4b6b-98eb-aa26853b3758\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " Jul 27 21:40:47.220386 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.220342 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q2g6\" (UniqueName: \"kubernetes.io/projected/991508b7-4c79-4b6b-98eb-aa26853b3758-kube-api-access-8q2g6\") pod \"991508b7-4c79-4b6b-98eb-aa26853b3758\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " Jul 27 21:40:47.220558 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.220388 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-util\") pod \"991508b7-4c79-4b6b-98eb-aa26853b3758\" (UID: \"991508b7-4c79-4b6b-98eb-aa26853b3758\") " Jul 27 21:40:47.220931 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.220901 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-bundle" (OuterVolumeSpecName: "bundle") pod "991508b7-4c79-4b6b-98eb-aa26853b3758" (UID: "991508b7-4c79-4b6b-98eb-aa26853b3758"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:47.222634 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.222606 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991508b7-4c79-4b6b-98eb-aa26853b3758-kube-api-access-8q2g6" (OuterVolumeSpecName: "kube-api-access-8q2g6") pod "991508b7-4c79-4b6b-98eb-aa26853b3758" (UID: "991508b7-4c79-4b6b-98eb-aa26853b3758"). InnerVolumeSpecName "kube-api-access-8q2g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:40:47.226009 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.225988 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-util" (OuterVolumeSpecName: "util") pod "991508b7-4c79-4b6b-98eb-aa26853b3758" (UID: "991508b7-4c79-4b6b-98eb-aa26853b3758"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:47.320901 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.320872 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-util\") pod \"b0004e05-a625-4091-bcdc-e21f0646f7e3\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " Jul 27 21:40:47.321061 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.320923 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-bundle\") pod \"b0004e05-a625-4091-bcdc-e21f0646f7e3\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " Jul 27 21:40:47.321061 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.320961 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdn79\" (UniqueName: \"kubernetes.io/projected/b0004e05-a625-4091-bcdc-e21f0646f7e3-kube-api-access-qdn79\") pod \"b0004e05-a625-4091-bcdc-e21f0646f7e3\" (UID: \"b0004e05-a625-4091-bcdc-e21f0646f7e3\") " Jul 27 21:40:47.321142 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.321089 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:47.321142 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.321099 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2g6\" (UniqueName: \"kubernetes.io/projected/991508b7-4c79-4b6b-98eb-aa26853b3758-kube-api-access-8q2g6\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:47.321142 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.321107 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991508b7-4c79-4b6b-98eb-aa26853b3758-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:47.321487 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.321459 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-bundle" (OuterVolumeSpecName: "bundle") pod "b0004e05-a625-4091-bcdc-e21f0646f7e3" (UID: "b0004e05-a625-4091-bcdc-e21f0646f7e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:47.323302 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.323270 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0004e05-a625-4091-bcdc-e21f0646f7e3-kube-api-access-qdn79" (OuterVolumeSpecName: "kube-api-access-qdn79") pod "b0004e05-a625-4091-bcdc-e21f0646f7e3" (UID: "b0004e05-a625-4091-bcdc-e21f0646f7e3"). InnerVolumeSpecName "kube-api-access-qdn79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:40:47.326433 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.326413 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-util" (OuterVolumeSpecName: "util") pod "b0004e05-a625-4091-bcdc-e21f0646f7e3" (UID: "b0004e05-a625-4091-bcdc-e21f0646f7e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:47.422473 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.422439 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:47.422473 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.422467 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0004e05-a625-4091-bcdc-e21f0646f7e3-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:47.422473 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.422480 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdn79\" (UniqueName: \"kubernetes.io/projected/b0004e05-a625-4091-bcdc-e21f0646f7e3-kube-api-access-qdn79\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:47.974583 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.974550 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" Jul 27 21:40:47.974583 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.974565 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4" event={"ID":"b0004e05-a625-4091-bcdc-e21f0646f7e3","Type":"ContainerDied","Data":"febe5f27e0616ab089eef5c31ae2dace25a218975ad89e9086e07b4fa7632823"} Jul 27 21:40:47.975086 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.974639 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="febe5f27e0616ab089eef5c31ae2dace25a218975ad89e9086e07b4fa7632823" Jul 27 21:40:47.976649 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.976624 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" Jul 27 21:40:47.976649 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.976632 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8" event={"ID":"991508b7-4c79-4b6b-98eb-aa26853b3758","Type":"ContainerDied","Data":"957627eb46d1e6c4dd6e53deb23042b30bb3ddc80a843d6443d3ede1dd06ef5e"} Jul 27 21:40:47.977003 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:47.976697 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="957627eb46d1e6c4dd6e53deb23042b30bb3ddc80a843d6443d3ede1dd06ef5e" Jul 27 21:40:48.024231 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.024191 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:48.115841 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.115818 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:48.128690 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.128647 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:48.142583 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.142562 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:48.195557 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.195525 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:48.228561 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.228479 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-bundle\") pod \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " Jul 27 21:40:48.228561 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.228518 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4flbs\" (UniqueName: \"kubernetes.io/projected/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-kube-api-access-4flbs\") pod \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " Jul 27 21:40:48.228808 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.228575 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-util\") pod \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\" (UID: \"4d37d026-6f4b-407e-9f9c-a65fb2fa777b\") " Jul 27 21:40:48.229048 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.229019 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-bundle" (OuterVolumeSpecName: "bundle") pod "4d37d026-6f4b-407e-9f9c-a65fb2fa777b" (UID: "4d37d026-6f4b-407e-9f9c-a65fb2fa777b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:48.231269 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.231235 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-kube-api-access-4flbs" (OuterVolumeSpecName: "kube-api-access-4flbs") pod "4d37d026-6f4b-407e-9f9c-a65fb2fa777b" (UID: "4d37d026-6f4b-407e-9f9c-a65fb2fa777b"). InnerVolumeSpecName "kube-api-access-4flbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:40:48.234452 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.234428 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-util" (OuterVolumeSpecName: "util") pod "4d37d026-6f4b-407e-9f9c-a65fb2fa777b" (UID: "4d37d026-6f4b-407e-9f9c-a65fb2fa777b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:48.329845 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.329809 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-bundle\") pod \"3290fdcd-a808-48ba-8d1f-be7e851e9546\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " Jul 27 21:40:48.330022 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.329867 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-util\") pod \"3290fdcd-a808-48ba-8d1f-be7e851e9546\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " Jul 27 21:40:48.330022 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.329932 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6szz\" (UniqueName: \"kubernetes.io/projected/3290fdcd-a808-48ba-8d1f-be7e851e9546-kube-api-access-s6szz\") pod \"3290fdcd-a808-48ba-8d1f-be7e851e9546\" (UID: \"3290fdcd-a808-48ba-8d1f-be7e851e9546\") " Jul 27 21:40:48.330143 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.330118 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:48.330197 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.330166 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4flbs\" (UniqueName: \"kubernetes.io/projected/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-kube-api-access-4flbs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:48.330197 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.330183 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d37d026-6f4b-407e-9f9c-a65fb2fa777b-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:48.330529 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.330502 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-bundle" (OuterVolumeSpecName: "bundle") pod "3290fdcd-a808-48ba-8d1f-be7e851e9546" (UID: "3290fdcd-a808-48ba-8d1f-be7e851e9546"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:48.332250 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.332229 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3290fdcd-a808-48ba-8d1f-be7e851e9546-kube-api-access-s6szz" (OuterVolumeSpecName: "kube-api-access-s6szz") pod "3290fdcd-a808-48ba-8d1f-be7e851e9546" (UID: "3290fdcd-a808-48ba-8d1f-be7e851e9546"). InnerVolumeSpecName "kube-api-access-s6szz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:40:48.335269 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.335247 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-util" (OuterVolumeSpecName: "util") pod "3290fdcd-a808-48ba-8d1f-be7e851e9546" (UID: "3290fdcd-a808-48ba-8d1f-be7e851e9546"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:40:48.430849 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.430797 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-util\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:48.430849 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.430847 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6szz\" (UniqueName: \"kubernetes.io/projected/3290fdcd-a808-48ba-8d1f-be7e851e9546-kube-api-access-s6szz\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:48.430849 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.430862 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3290fdcd-a808-48ba-8d1f-be7e851e9546-bundle\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:40:48.983565 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.983527 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" Jul 27 21:40:48.983565 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.983543 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr" event={"ID":"3290fdcd-a808-48ba-8d1f-be7e851e9546","Type":"ContainerDied","Data":"eae2bae18e34a6d7f4ca03401bfb2d0b8d88da496d482ca5d4654e37ff6144ef"} Jul 27 21:40:48.984049 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.983589 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae2bae18e34a6d7f4ca03401bfb2d0b8d88da496d482ca5d4654e37ff6144ef" Jul 27 21:40:48.985407 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.985383 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" Jul 27 21:40:48.985407 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.985398 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk" event={"ID":"4d37d026-6f4b-407e-9f9c-a65fb2fa777b","Type":"ContainerDied","Data":"2de99a5535fad0411784e13293abef788c70f4ff5ffdf0afd7bac828edad131e"} Jul 27 21:40:48.985555 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:48.985435 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de99a5535fad0411784e13293abef788c70f4ff5ffdf0afd7bac828edad131e" Jul 27 21:40:50.023691 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:50.023631 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:50.607461 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:50.607421 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:51.685764 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:51.685731 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:53.807930 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.807899 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8"] Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808188 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerName="pull" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808198 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerName="pull" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808213 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808218 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808225 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerName="extract" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808232 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerName="extract" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808239 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808244 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808249 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerName="pull" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808253 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerName="pull" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808259 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerName="extract" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808264 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerName="extract" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808271 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808276 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808280 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerName="extract" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808285 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerName="extract" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808291 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808295 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerName="util" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808301 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerName="pull" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808305 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerName="pull" Jul 27 21:40:53.808307 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808313 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerName="pull" Jul 27 21:40:53.808890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808319 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerName="pull" Jul 27 21:40:53.808890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808326 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerName="extract" Jul 27 21:40:53.808890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808330 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerName="extract" Jul 27 21:40:53.808890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808379 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3290fdcd-a808-48ba-8d1f-be7e851e9546" containerName="extract" Jul 27 21:40:53.808890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808385 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="991508b7-4c79-4b6b-98eb-aa26853b3758" containerName="extract" Jul 27 21:40:53.808890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808392 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0004e05-a625-4091-bcdc-e21f0646f7e3" containerName="extract" Jul 27 21:40:53.808890 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.808397 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d37d026-6f4b-407e-9f9c-a65fb2fa777b" containerName="extract" Jul 27 21:40:53.812416 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.812401 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" Jul 27 21:40:53.818016 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.817997 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w2wtm\"" Jul 27 21:40:53.828552 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.828528 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8"] Jul 27 21:40:53.868336 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.868303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwrlc\" (UniqueName: \"kubernetes.io/projected/7eaed7ec-8d14-44c8-b611-01c31323f2ab-kube-api-access-gwrlc\") pod \"authorino-operator-75cc8d5cb7-mgvg8\" (UID: \"7eaed7ec-8d14-44c8-b611-01c31323f2ab\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" Jul 27 21:40:53.969284 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.969247 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwrlc\" (UniqueName: \"kubernetes.io/projected/7eaed7ec-8d14-44c8-b611-01c31323f2ab-kube-api-access-gwrlc\") pod \"authorino-operator-75cc8d5cb7-mgvg8\" (UID: \"7eaed7ec-8d14-44c8-b611-01c31323f2ab\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" Jul 27 21:40:53.981873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:53.981844 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwrlc\" (UniqueName: \"kubernetes.io/projected/7eaed7ec-8d14-44c8-b611-01c31323f2ab-kube-api-access-gwrlc\") pod \"authorino-operator-75cc8d5cb7-mgvg8\" (UID: \"7eaed7ec-8d14-44c8-b611-01c31323f2ab\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" Jul 27 21:40:54.122412 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:54.122304 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" Jul 27 21:40:54.279911 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:54.279875 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8"] Jul 27 21:40:54.285389 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:54.285358 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaed7ec_8d14_44c8_b611_01c31323f2ab.slice/crio-3efc692958368b2b607ea0fb36fda78714c0159fa0a087db2c850bacfddbb1bf WatchSource:0}: Error finding container 3efc692958368b2b607ea0fb36fda78714c0159fa0a087db2c850bacfddbb1bf: Status 404 returned error can't find the container with id 3efc692958368b2b607ea0fb36fda78714c0159fa0a087db2c850bacfddbb1bf Jul 27 21:40:54.889045 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:54.889012 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:55.015829 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:55.015778 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" event={"ID":"7eaed7ec-8d14-44c8-b611-01c31323f2ab","Type":"ContainerStarted","Data":"3efc692958368b2b607ea0fb36fda78714c0159fa0a087db2c850bacfddbb1bf"} Jul 27 21:40:56.944321 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:56.944280 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx"] Jul 27 21:40:56.947767 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:56.947745 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" Jul 27 21:40:56.950587 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:56.950562 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-swszf\"" Jul 27 21:40:56.952421 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:56.952406 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 27 21:40:56.963337 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:56.963312 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx"] Jul 27 21:40:56.991057 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:56.991024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbff\" (UniqueName: \"kubernetes.io/projected/2b404ec0-9d1a-40b8-9e41-549f4f4b5b59-kube-api-access-9wbff\") pod \"dns-operator-controller-manager-864dd78c4c-pdzgx\" (UID: \"2b404ec0-9d1a-40b8-9e41-549f4f4b5b59\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" Jul 27 21:40:57.028714 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.028652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" event={"ID":"7eaed7ec-8d14-44c8-b611-01c31323f2ab","Type":"ContainerStarted","Data":"2988cda0a1113b2b080fc416becda4f18f9ed19a469a283cc982dc591e71fd7f"} Jul 27 21:40:57.028894 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.028748 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" Jul 27 21:40:57.053088 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.053033 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" podStartSLOduration=1.9005868989999999 podStartE2EDuration="4.053016384s" totalImagesPullingTime="2.152429485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:57.050355894 +0000 UTC m=+425.458777924" watchObservedRunningTime="2026-07-27 21:40:57.053016384 +0000 UTC m=+425.461438414" Jul 27 21:40:57.092524 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.092490 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbff\" (UniqueName: \"kubernetes.io/projected/2b404ec0-9d1a-40b8-9e41-549f4f4b5b59-kube-api-access-9wbff\") pod \"dns-operator-controller-manager-864dd78c4c-pdzgx\" (UID: \"2b404ec0-9d1a-40b8-9e41-549f4f4b5b59\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" Jul 27 21:40:57.104052 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.104018 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbff\" (UniqueName: \"kubernetes.io/projected/2b404ec0-9d1a-40b8-9e41-549f4f4b5b59-kube-api-access-9wbff\") pod \"dns-operator-controller-manager-864dd78c4c-pdzgx\" (UID: \"2b404ec0-9d1a-40b8-9e41-549f4f4b5b59\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" Jul 27 21:40:57.258088 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.257978 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" Jul 27 21:40:57.401105 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.401075 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx"] Jul 27 21:40:57.402992 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:40:57.402962 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b404ec0_9d1a_40b8_9e41_549f4f4b5b59.slice/crio-db67c40cbfd2d41863ef6398c875c55ec7ed64c096ff726ef4f1f38b5564f340 WatchSource:0}: Error finding container db67c40cbfd2d41863ef6398c875c55ec7ed64c096ff726ef4f1f38b5564f340: Status 404 returned error can't find the container with id db67c40cbfd2d41863ef6398c875c55ec7ed64c096ff726ef4f1f38b5564f340 Jul 27 21:40:57.637056 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.636966 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:57.893909 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:57.893826 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:40:58.036005 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:58.035970 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" event={"ID":"2b404ec0-9d1a-40b8-9e41-549f4f4b5b59","Type":"ContainerStarted","Data":"db67c40cbfd2d41863ef6398c875c55ec7ed64c096ff726ef4f1f38b5564f340"} Jul 27 21:40:59.692877 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:59.692841 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv"] Jul 27 21:40:59.696046 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:59.696029 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:40:59.698932 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:59.698908 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d7lcq\"" Jul 27 21:40:59.708424 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:59.708401 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv"] Jul 27 21:40:59.715918 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:59.715894 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv6kc\" (UniqueName: \"kubernetes.io/projected/6a900478-f4a0-4e78-99f5-08952daa82ee-kube-api-access-mv6kc\") pod \"limitador-operator-controller-manager-859d7b78bd-zsncv\" (UID: \"6a900478-f4a0-4e78-99f5-08952daa82ee\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:40:59.816887 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:59.816846 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv6kc\" (UniqueName: \"kubernetes.io/projected/6a900478-f4a0-4e78-99f5-08952daa82ee-kube-api-access-mv6kc\") pod \"limitador-operator-controller-manager-859d7b78bd-zsncv\" (UID: \"6a900478-f4a0-4e78-99f5-08952daa82ee\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:40:59.830141 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:40:59.830101 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv6kc\" (UniqueName: \"kubernetes.io/projected/6a900478-f4a0-4e78-99f5-08952daa82ee-kube-api-access-mv6kc\") pod \"limitador-operator-controller-manager-859d7b78bd-zsncv\" (UID: \"6a900478-f4a0-4e78-99f5-08952daa82ee\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:00.005662 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:00.005621 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:00.048330 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:00.048292 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" event={"ID":"2b404ec0-9d1a-40b8-9e41-549f4f4b5b59","Type":"ContainerStarted","Data":"bdb2343ad8384dbf259d7d71baf5c3a358dc0f383ee377f65107ef7c4448d440"} Jul 27 21:41:00.048501 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:00.048418 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" Jul 27 21:41:00.073511 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:00.073460 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" podStartSLOduration=2.2915185 podStartE2EDuration="4.073442039s" totalImagesPullingTime="1.781923539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:41:00.07060525 +0000 UTC m=+428.479027292" watchObservedRunningTime="2026-07-27 21:41:00.073442039 +0000 UTC m=+428.481864069" Jul 27 21:41:00.097506 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:00.097461 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:00.148062 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:00.148036 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv"] Jul 27 21:41:00.150440 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:41:00.150406 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a900478_f4a0_4e78_99f5_08952daa82ee.slice/crio-14fb2453b6616bd4e0a4a2694eca741f6a9f71cda9a002d2a368a0eba5364b9c WatchSource:0}: Error finding container 14fb2453b6616bd4e0a4a2694eca741f6a9f71cda9a002d2a368a0eba5364b9c: Status 404 returned error can't find the container with id 14fb2453b6616bd4e0a4a2694eca741f6a9f71cda9a002d2a368a0eba5364b9c Jul 27 21:41:00.227758 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:00.227717 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:01.055293 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:01.055247 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" event={"ID":"6a900478-f4a0-4e78-99f5-08952daa82ee","Type":"ContainerStarted","Data":"14fb2453b6616bd4e0a4a2694eca741f6a9f71cda9a002d2a368a0eba5364b9c"} Jul 27 21:41:02.061707 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:02.061584 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" event={"ID":"6a900478-f4a0-4e78-99f5-08952daa82ee","Type":"ContainerStarted","Data":"b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5"} Jul 27 21:41:02.061707 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:02.061694 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:02.078869 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:02.078818 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" podStartSLOduration=1.456748922 podStartE2EDuration="3.078799036s" totalImagesPullingTime="1.622050114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:41:02.077616023 +0000 UTC m=+430.486038065" watchObservedRunningTime="2026-07-27 21:41:02.078799036 +0000 UTC m=+430.487221134" Jul 27 21:41:02.929078 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:02.929041 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:04.083797 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:04.083766 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:04.277460 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:04.277418 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:05.423418 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:05.423376 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:05.527123 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:05.527083 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:05.851174 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:05.851139 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:06.082712 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:06.082661 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:08.038833 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:08.038805 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mgvg8" Jul 27 21:41:09.622370 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:09.622337 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:09.724033 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:09.723996 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:11.058032 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:11.058005 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdzgx" Jul 27 21:41:13.069627 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.069548 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:13.429137 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.429041 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz"] Jul 27 21:41:13.432185 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.432160 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.435879 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.435857 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vx4tb\"" Jul 27 21:41:13.436148 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.436130 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 27 21:41:13.436517 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.436501 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 27 21:41:13.445553 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.445527 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:13.459711 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.457251 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz"] Jul 27 21:41:13.529622 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.529581 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34cc2ea3-47d9-4416-bea6-72011567f324-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.529622 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.529618 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftz5\" (UniqueName: \"kubernetes.io/projected/34cc2ea3-47d9-4416-bea6-72011567f324-kube-api-access-wftz5\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.529883 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.529755 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/34cc2ea3-47d9-4416-bea6-72011567f324-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.630410 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.630374 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34cc2ea3-47d9-4416-bea6-72011567f324-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.630410 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.630408 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftz5\" (UniqueName: \"kubernetes.io/projected/34cc2ea3-47d9-4416-bea6-72011567f324-kube-api-access-wftz5\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.630638 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.630477 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/34cc2ea3-47d9-4416-bea6-72011567f324-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.631104 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.631083 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34cc2ea3-47d9-4416-bea6-72011567f324-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.633043 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.633025 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/34cc2ea3-47d9-4416-bea6-72011567f324-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.654121 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.654085 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftz5\" (UniqueName: \"kubernetes.io/projected/34cc2ea3-47d9-4416-bea6-72011567f324-kube-api-access-wftz5\") pod \"kuadrant-console-plugin-66b5774985-vxtpz\" (UID: \"34cc2ea3-47d9-4416-bea6-72011567f324\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.690553 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.690457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:13.743254 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.743217 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" Jul 27 21:41:13.791048 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.791003 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:13.884353 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:13.884331 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz"] Jul 27 21:41:13.886109 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:41:13.886083 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34cc2ea3_47d9_4416_bea6_72011567f324.slice/crio-5dc05a5cd11c8d76816c82e644c5e156c069137b8a21161220ea38679c905ea6 WatchSource:0}: Error finding container 5dc05a5cd11c8d76816c82e644c5e156c069137b8a21161220ea38679c905ea6: Status 404 returned error can't find the container with id 5dc05a5cd11c8d76816c82e644c5e156c069137b8a21161220ea38679c905ea6 Jul 27 21:41:14.122542 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:14.122500 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" event={"ID":"34cc2ea3-47d9-4416-bea6-72011567f324","Type":"ContainerStarted","Data":"5dc05a5cd11c8d76816c82e644c5e156c069137b8a21161220ea38679c905ea6"} Jul 27 21:41:16.991188 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:16.991154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:17.092852 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:17.092820 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:23.908467 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.907737 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv"] Jul 27 21:41:23.908467 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.907962 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" containerName="manager" containerID="cri-o://b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5" gracePeriod=2 Jul 27 21:41:23.918861 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.918821 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv"] Jul 27 21:41:23.930026 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:41:23.929984 2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"6a900478-f4a0-4e78-99f5-08952daa82ee\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-zsncv\": pods \"limitador-operator-controller-manager-859d7b78bd-zsncv\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:23.941863 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.941804 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg"] Jul 27 21:41:23.942703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.942307 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" containerName="manager" Jul 27 21:41:23.942703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.942332 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" containerName="manager" Jul 27 21:41:23.942703 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.942423 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" containerName="manager" Jul 27 21:41:23.945770 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.945748 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" Jul 27 21:41:23.960922 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:23.960891 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg"] Jul 27 21:41:23.979215 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:41:23.979177 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zsncv\" is forbidden: User \"system:node:ip-10-0-133-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-37.ec2.internal' and this object" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:24.025489 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:24.025453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2pz\" (UniqueName: \"kubernetes.io/projected/41464d90-b6cb-461e-8ecb-89de3caf27dc-kube-api-access-sd2pz\") pod \"limitador-operator-controller-manager-859d7b78bd-kh6jg\" (UID: \"41464d90-b6cb-461e-8ecb-89de3caf27dc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" Jul 27 21:41:24.127047 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:24.127002 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2pz\" (UniqueName: \"kubernetes.io/projected/41464d90-b6cb-461e-8ecb-89de3caf27dc-kube-api-access-sd2pz\") pod \"limitador-operator-controller-manager-859d7b78bd-kh6jg\" (UID: \"41464d90-b6cb-461e-8ecb-89de3caf27dc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" Jul 27 21:41:24.136793 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:24.136756 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2pz\" (UniqueName: \"kubernetes.io/projected/41464d90-b6cb-461e-8ecb-89de3caf27dc-kube-api-access-sd2pz\") pod \"limitador-operator-controller-manager-859d7b78bd-kh6jg\" (UID: \"41464d90-b6cb-461e-8ecb-89de3caf27dc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" Jul 27 21:41:24.283716 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:24.283659 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" Jul 27 21:41:24.941001 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:24.940938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:25.043973 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:25.043930 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:25.286477 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:25.286444 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:25.386129 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:25.386093 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:25.593123 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:25.593035 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:27.656741 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:27.656704 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:27.757409 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:27.757368 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:31.869205 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:31.869163 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:31.970008 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:31.969971 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:35.462449 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:35.462415 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:35.565404 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:35.565363 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:36.478211 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:36.478192 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:36.480462 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:41:36.480439 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zsncv\" is forbidden: User \"system:node:ip-10-0-133-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-37.ec2.internal' and this object" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:36.484826 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:36.484805 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg"] Jul 27 21:41:36.486939 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:41:36.486909 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41464d90_b6cb_461e_8ecb_89de3caf27dc.slice/crio-3a47209ce397987c3802d9d4adde4bd3a355fdecd39c6eea2f79d3500db91dc3 WatchSource:0}: Error finding container 3a47209ce397987c3802d9d4adde4bd3a355fdecd39c6eea2f79d3500db91dc3: Status 404 returned error can't find the container with id 3a47209ce397987c3802d9d4adde4bd3a355fdecd39c6eea2f79d3500db91dc3 Jul 27 21:41:36.644738 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:36.644634 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv6kc\" (UniqueName: \"kubernetes.io/projected/6a900478-f4a0-4e78-99f5-08952daa82ee-kube-api-access-mv6kc\") pod \"6a900478-f4a0-4e78-99f5-08952daa82ee\" (UID: \"6a900478-f4a0-4e78-99f5-08952daa82ee\") " Jul 27 21:41:36.646853 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:36.646829 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a900478-f4a0-4e78-99f5-08952daa82ee-kube-api-access-mv6kc" (OuterVolumeSpecName: "kube-api-access-mv6kc") pod "6a900478-f4a0-4e78-99f5-08952daa82ee" (UID: "6a900478-f4a0-4e78-99f5-08952daa82ee"). InnerVolumeSpecName "kube-api-access-mv6kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:41:36.746052 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:36.746000 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv6kc\" (UniqueName: \"kubernetes.io/projected/6a900478-f4a0-4e78-99f5-08952daa82ee-kube-api-access-mv6kc\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:41:37.254710 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.254662 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" event={"ID":"41464d90-b6cb-461e-8ecb-89de3caf27dc","Type":"ContainerStarted","Data":"db6fea3deba77d7a2c889b8f82a8d780a9e58758326eb9bed95f656f1a40ea4b"} Jul 27 21:41:37.254895 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.254737 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" event={"ID":"41464d90-b6cb-461e-8ecb-89de3caf27dc","Type":"ContainerStarted","Data":"3a47209ce397987c3802d9d4adde4bd3a355fdecd39c6eea2f79d3500db91dc3"} Jul 27 21:41:37.254895 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.254768 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" Jul 27 21:41:37.256323 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.256294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" event={"ID":"34cc2ea3-47d9-4416-bea6-72011567f324","Type":"ContainerStarted","Data":"44dac847f6ffee5aca979e2f20d7e8a27d432f807690dec0d31edda341456511"} Jul 27 21:41:37.257589 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.257561 2556 generic.go:356] "Generic (PLEG): container finished" podID="6a900478-f4a0-4e78-99f5-08952daa82ee" containerID="b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5" exitCode=0 Jul 27 21:41:37.257741 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.257594 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:37.257741 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.257665 2556 scope.go:117] "RemoveContainer" containerID="b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5" Jul 27 21:41:37.266648 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.266629 2556 scope.go:117] "RemoveContainer" containerID="b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5" Jul 27 21:41:37.266958 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:41:37.266939 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5\": container with ID starting with b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5 not found: ID does not exist" containerID="b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5" Jul 27 21:41:37.267012 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.266968 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5"} err="failed to get container status \"b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5\": rpc error: code = NotFound desc = could not find container \"b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5\": container with ID starting with b22bb659fc2f4c8198a32ba0528dc0e73cc53ef6438a6e261b254a58bd7b9ca5 not found: ID does not exist" Jul 27 21:41:37.289068 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.289026 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" podStartSLOduration=14.289013069 podStartE2EDuration="14.289013069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:41:37.286529209 +0000 UTC m=+465.694951238" watchObservedRunningTime="2026-07-27 21:41:37.289013069 +0000 UTC m=+465.697435098" Jul 27 21:41:37.289305 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:41:37.289286 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zsncv\" is forbidden: User \"system:node:ip-10-0-133-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-37.ec2.internal' and this object" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:37.291680 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:41:37.291649 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zsncv\" is forbidden: User \"system:node:ip-10-0-133-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-37.ec2.internal' and this object" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zsncv" Jul 27 21:41:37.334563 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:37.334488 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vxtpz" podStartSLOduration=1.818611443 podStartE2EDuration="24.334467296s" totalImagesPullingTime="22.515855853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:41:37.333210161 +0000 UTC m=+465.741632192" watchObservedRunningTime="2026-07-27 21:41:37.334467296 +0000 UTC m=+465.742889326" Jul 27 21:41:38.193050 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:38.193020 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a900478-f4a0-4e78-99f5-08952daa82ee" path="/var/lib/kubelet/pods/6a900478-f4a0-4e78-99f5-08952daa82ee/volumes" Jul 27 21:41:48.267089 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:48.267055 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kh6jg" Jul 27 21:41:52.597542 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.597512 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp"] Jul 27 21:41:52.624221 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.624181 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:52.743719 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.743650 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp"] Jul 27 21:41:52.743911 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.743804 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.746773 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.746744 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-247w6\"" Jul 27 21:41:52.780154 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780120 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780330 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780181 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780330 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780219 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780330 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780330 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780316 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780564 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780371 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780564 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780396 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db631f7d-1a1a-4446-a872-9c91fb4270bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780564 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780427 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxtlv\" (UniqueName: \"kubernetes.io/projected/db631f7d-1a1a-4446-a872-9c91fb4270bf-kube-api-access-wxtlv\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.780564 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.780453 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.856358 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.856264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:52.881392 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881363 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881561 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881396 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db631f7d-1a1a-4446-a872-9c91fb4270bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881561 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881416 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtlv\" (UniqueName: \"kubernetes.io/projected/db631f7d-1a1a-4446-a872-9c91fb4270bf-kube-api-access-wxtlv\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881658 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881584 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881658 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881796 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881796 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881777 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881901 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881830 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881901 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881862 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.881901 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.881877 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.882217 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.882187 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.882345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.882241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.882345 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.882271 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.882641 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.882537 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db631f7d-1a1a-4446-a872-9c91fb4270bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.884739 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.884713 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.891686 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.891646 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtlv\" (UniqueName: \"kubernetes.io/projected/db631f7d-1a1a-4446-a872-9c91fb4270bf-kube-api-access-wxtlv\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.891843 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.891757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:52.894482 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:52.894461 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db631f7d-1a1a-4446-a872-9c91fb4270bf-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-xp6zp\" (UID: \"db631f7d-1a1a-4446-a872-9c91fb4270bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:53.055104 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:53.055063 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:53.226509 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:53.226478 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp"] Jul 27 21:41:53.227560 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:41:53.227537 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb631f7d_1a1a_4446_a872_9c91fb4270bf.slice/crio-3cc8c2812be685d2fab7bbfc469f07161e58c8bda6574d89b62605355540c2ec WatchSource:0}: Error finding container 3cc8c2812be685d2fab7bbfc469f07161e58c8bda6574d89b62605355540c2ec: Status 404 returned error can't find the container with id 3cc8c2812be685d2fab7bbfc469f07161e58c8bda6574d89b62605355540c2ec Jul 27 21:41:53.230213 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:53.230177 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:41:53.230295 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:53.230245 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:41:53.230295 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:53.230273 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:41:53.344738 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:53.344707 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" event={"ID":"db631f7d-1a1a-4446-a872-9c91fb4270bf","Type":"ContainerStarted","Data":"3cc8c2812be685d2fab7bbfc469f07161e58c8bda6574d89b62605355540c2ec"} Jul 27 21:41:54.351250 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:54.351205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" event={"ID":"db631f7d-1a1a-4446-a872-9c91fb4270bf","Type":"ContainerStarted","Data":"09d69e368ad9550b1226f48dc66e59fe2993d58c70baa991eff57f2ad4f8e1d8"} Jul 27 21:41:54.374040 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:54.373992 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" podStartSLOduration=2.373973596 podStartE2EDuration="2.373973596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:41:54.372266065 +0000 UTC m=+482.780688086" watchObservedRunningTime="2026-07-27 21:41:54.373973596 +0000 UTC m=+482.782395625" Jul 27 21:41:55.055571 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:55.055537 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:55.060356 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:55.060329 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:55.356071 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:55.355989 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:55.357148 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:55.357126 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:41:58.024983 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:58.024945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:41:58.080543 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:41:58.080513 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:03.579718 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:03.579685 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:03.678468 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:03.678433 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:03.737465 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:03.737411 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:04.002441 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.002408 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:42:04.083898 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.083858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:42:04.083898 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.083899 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.084094 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.083924 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:42:04.086548 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.086524 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 21:42:04.176405 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.176363 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8dh8\" (UniqueName: \"kubernetes.io/projected/55aad1c8-4a1b-425f-94ed-0842980f3a83-kube-api-access-r8dh8\") pod \"limitador-limitador-54d6d988d5-vvwp4\" (UID: \"55aad1c8-4a1b-425f-94ed-0842980f3a83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.176405 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.176409 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55aad1c8-4a1b-425f-94ed-0842980f3a83-config-file\") pod \"limitador-limitador-54d6d988d5-vvwp4\" (UID: \"55aad1c8-4a1b-425f-94ed-0842980f3a83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.277328 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.277246 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8dh8\" (UniqueName: \"kubernetes.io/projected/55aad1c8-4a1b-425f-94ed-0842980f3a83-kube-api-access-r8dh8\") pod \"limitador-limitador-54d6d988d5-vvwp4\" (UID: \"55aad1c8-4a1b-425f-94ed-0842980f3a83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.277328 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.277287 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55aad1c8-4a1b-425f-94ed-0842980f3a83-config-file\") pod \"limitador-limitador-54d6d988d5-vvwp4\" (UID: \"55aad1c8-4a1b-425f-94ed-0842980f3a83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.277918 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.277900 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/55aad1c8-4a1b-425f-94ed-0842980f3a83-config-file\") pod \"limitador-limitador-54d6d988d5-vvwp4\" (UID: \"55aad1c8-4a1b-425f-94ed-0842980f3a83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.286164 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.286138 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8dh8\" (UniqueName: \"kubernetes.io/projected/55aad1c8-4a1b-425f-94ed-0842980f3a83-kube-api-access-r8dh8\") pod \"limitador-limitador-54d6d988d5-vvwp4\" (UID: \"55aad1c8-4a1b-425f-94ed-0842980f3a83\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.394575 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.394538 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:04.428089 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.428060 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:04.503376 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.503346 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:04.541304 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:04.541280 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:42:04.543963 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:42:04.543933 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55aad1c8_4a1b_425f_94ed_0842980f3a83.slice/crio-35edb77a6c15a78405992acdae0467e071ba2be0a91075587e2af03edd173356 WatchSource:0}: Error finding container 35edb77a6c15a78405992acdae0467e071ba2be0a91075587e2af03edd173356: Status 404 returned error can't find the container with id 35edb77a6c15a78405992acdae0467e071ba2be0a91075587e2af03edd173356 Jul 27 21:42:05.408851 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:05.408807 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" event={"ID":"55aad1c8-4a1b-425f-94ed-0842980f3a83","Type":"ContainerStarted","Data":"35edb77a6c15a78405992acdae0467e071ba2be0a91075587e2af03edd173356"} Jul 27 21:42:08.425945 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:08.425910 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" event={"ID":"55aad1c8-4a1b-425f-94ed-0842980f3a83","Type":"ContainerStarted","Data":"010d9d0b9ef32d059fd639d0ee6ce4d872156c06551edfa03ec837627ed21ca2"} Jul 27 21:42:08.426337 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:08.426011 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:08.446699 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:08.446581 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" podStartSLOduration=2.550996531 podStartE2EDuration="5.446563568s" totalImagesPullingTime="2.895567037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:42:08.446449446 +0000 UTC m=+496.854871475" watchObservedRunningTime="2026-07-27 21:42:08.446563568 +0000 UTC m=+496.854985599" Jul 27 21:42:16.843653 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:16.843614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:19.432046 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:19.432017 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vvwp4" Jul 27 21:42:21.984378 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:21.984339 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:22.066967 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:22.066930 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:22.473970 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:22.473941 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:34.374221 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:34.374179 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:34.859369 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:34.859323 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:36.257585 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:36.257548 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:36.353817 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:36.353787 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:41.441319 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:41.441267 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:41.538999 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:41.538942 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:41.603417 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:41.603382 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:44.567076 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:44.566985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:42:44.664041 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:42:44.663998 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:16.157383 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:16.157342 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:16.247947 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:16.247909 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:16.758504 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:16.758453 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:17.272467 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:17.272434 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:17.370538 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:17.370508 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:17.487852 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:17.487808 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:17.533526 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:17.533442 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:43:25.357929 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.357828 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xp6zp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 27 21:43:25.357929 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.357902 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" podUID="db631f7d-1a1a-4446-a872-9c91fb4270bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 27 21:43:25.813572 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.813537 2556 generic.go:356] "Generic (PLEG): container finished" podID="db631f7d-1a1a-4446-a872-9c91fb4270bf" containerID="09d69e368ad9550b1226f48dc66e59fe2993d58c70baa991eff57f2ad4f8e1d8" exitCode=0 Jul 27 21:43:25.813803 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.813616 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" event={"ID":"db631f7d-1a1a-4446-a872-9c91fb4270bf","Type":"ContainerDied","Data":"09d69e368ad9550b1226f48dc66e59fe2993d58c70baa991eff57f2ad4f8e1d8"} Jul 27 21:43:25.814029 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.814015 2556 scope.go:117] "RemoveContainer" containerID="09d69e368ad9550b1226f48dc66e59fe2993d58c70baa991eff57f2ad4f8e1d8" Jul 27 21:43:25.814998 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.814971 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:43:25.815052 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.815038 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:43:25.815087 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:25.815069 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 21:43:26.825314 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:26.825265 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" event={"ID":"db631f7d-1a1a-4446-a872-9c91fb4270bf","Type":"ContainerStarted","Data":"f65f8e0be1c6e076f8cb98568d8704c8cbf0bf465ec9c30ad4e37de865eeb67d"} Jul 27 21:43:27.056315 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:27.056275 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:43:27.056533 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:27.056511 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xp6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 27 21:43:27.056584 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:27.056564 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" podUID="db631f7d-1a1a-4446-a872-9c91fb4270bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 27 21:43:27.309145 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:27.309104 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:27.406864 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:27.406825 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:28.056264 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:28.056230 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xp6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 27 21:43:28.056709 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:28.056287 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" podUID="db631f7d-1a1a-4446-a872-9c91fb4270bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 27 21:43:29.056354 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:29.056316 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xp6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 27 21:43:29.056779 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:29.056392 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" podUID="db631f7d-1a1a-4446-a872-9c91fb4270bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 27 21:43:30.055946 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:30.055904 2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-xp6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 27 21:43:30.056156 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:30.055969 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" podUID="db631f7d-1a1a-4446-a872-9c91fb4270bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 27 21:43:31.059365 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:31.059333 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:43:31.059855 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:31.059703 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:43:31.060401 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:31.060380 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-xp6zp" Jul 27 21:43:32.082714 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:32.082663 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:32.589085 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:32.589042 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:43:47.120826 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:47.120786 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:43:47.998941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:47.998913 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:43:59.201451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:43:59.201414 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:44:00.405500 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:00.405467 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:44:00.572696 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:00.572632 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:44:10.466124 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:10.466090 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:44:12.205835 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:12.205732 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:44:13.087686 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.087643 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt"] Jul 27 21:44:13.090256 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.090238 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.094082 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.094046 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 27 21:44:13.094221 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.094113 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dwq99\"" Jul 27 21:44:13.094221 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.094060 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 21:44:13.094346 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.094271 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 21:44:13.105166 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.105141 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt"] Jul 27 21:44:13.115756 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.115720 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:44:13.173544 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.173510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.173544 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.173547 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jscw\" (UniqueName: \"kubernetes.io/projected/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-kube-api-access-5jscw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.173806 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.173594 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.173806 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.173620 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.173806 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.173638 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.173806 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.173710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.274720 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.274640 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.274720 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.274734 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.275239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.274755 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jscw\" (UniqueName: \"kubernetes.io/projected/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-kube-api-access-5jscw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.275239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.274796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.275239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.274820 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.275239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.274837 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.275239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.275170 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.275239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.275167 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.275490 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.275277 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.277222 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.277201 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.277442 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.277426 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.283308 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.283282 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jscw\" (UniqueName: \"kubernetes.io/projected/2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd-kube-api-access-5jscw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-97bzt\" (UID: \"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.401060 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.400962 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:13.571128 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.571099 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt"] Jul 27 21:44:13.573289 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:44:13.573261 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea6bc5c_e748_4bd6_9ca7_118375dc8dbd.slice/crio-583b708a95357f2c82233de11bf244aa505abf1bdb2197a32727ae0d74df104f WatchSource:0}: Error finding container 583b708a95357f2c82233de11bf244aa505abf1bdb2197a32727ae0d74df104f: Status 404 returned error can't find the container with id 583b708a95357f2c82233de11bf244aa505abf1bdb2197a32727ae0d74df104f Jul 27 21:44:13.574633 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:13.574610 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 21:44:14.074716 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:14.074626 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" event={"ID":"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd","Type":"ContainerStarted","Data":"583b708a95357f2c82233de11bf244aa505abf1bdb2197a32727ae0d74df104f"} Jul 27 21:44:15.081938 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:15.081903 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" event={"ID":"2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd","Type":"ContainerStarted","Data":"6647f594d163c227920627672954720c487ff8c4049fe0141401e5d8cc935161"} Jul 27 21:44:15.101533 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:15.101479 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" podStartSLOduration=1.199103285 podStartE2EDuration="2.101462136s" totalImagesPullingTime="902.358851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:44:15.099451081 +0000 UTC m=+623.507873125" watchObservedRunningTime="2026-07-27 21:44:15.101462136 +0000 UTC m=+623.509884166" Jul 27 21:44:17.105250 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.105216 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d"] Jul 27 21:44:17.107940 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.107923 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.113176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.113153 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 27 21:44:17.122821 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.122793 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d"] Jul 27 21:44:17.174023 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.173992 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:44:17.213070 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.213036 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.213070 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.213074 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.213281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.213094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.213281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.213154 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhzl\" (UniqueName: \"kubernetes.io/projected/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-kube-api-access-bmhzl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.213281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.213197 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.213281 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.213229 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.313957 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.313923 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.313957 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.313961 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.314180 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.314001 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.314180 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.314065 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhzl\" (UniqueName: \"kubernetes.io/projected/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-kube-api-access-bmhzl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.314461 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.314369 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.314461 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.314429 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.314729 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.314491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.314729 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.314545 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.315123 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.315083 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.316736 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.316709 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.317251 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.317226 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.323048 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.323026 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhzl\" (UniqueName: \"kubernetes.io/projected/c3a3f640-b2a6-4d63-9b3a-efea5401f9e1-kube-api-access-bmhzl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d\" (UID: \"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.417855 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.417774 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:17.563040 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:17.563007 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d"] Jul 27 21:44:17.564740 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:44:17.564714 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3a3f640_b2a6_4d63_9b3a_efea5401f9e1.slice/crio-0ca4a7cefeba580780388bb3f2c3a16a22a4dd4ecfbacec142cc87c82b18cfae WatchSource:0}: Error finding container 0ca4a7cefeba580780388bb3f2c3a16a22a4dd4ecfbacec142cc87c82b18cfae: Status 404 returned error can't find the container with id 0ca4a7cefeba580780388bb3f2c3a16a22a4dd4ecfbacec142cc87c82b18cfae Jul 27 21:44:18.100428 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:18.100330 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" event={"ID":"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1","Type":"ContainerStarted","Data":"c2980c4077815c52e73e83484468011df8111f85df17703cfde05df59d058397"} Jul 27 21:44:18.100428 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:18.100382 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" event={"ID":"c3a3f640-b2a6-4d63-9b3a-efea5401f9e1","Type":"ContainerStarted","Data":"0ca4a7cefeba580780388bb3f2c3a16a22a4dd4ecfbacec142cc87c82b18cfae"} Jul 27 21:44:18.123537 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:18.123484 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" podStartSLOduration=0.958279012 podStartE2EDuration="1.123468287s" totalImagesPullingTime="165.189275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:44:18.121102415 +0000 UTC m=+626.529524445" watchObservedRunningTime="2026-07-27 21:44:18.123468287 +0000 UTC m=+626.531890316" Jul 27 21:44:22.285986 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:22.285949 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:44:23.401082 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:23.401046 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:23.401485 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:23.401108 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:23.413948 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:23.413922 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:24.142860 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:24.142828 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-97bzt" Jul 27 21:44:27.418998 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:27.418965 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:27.419577 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:27.419015 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:27.431745 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:27.431721 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:28.162545 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:28.162514 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d" Jul 27 21:44:32.839265 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:32.839225 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:44:32.937215 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:32.937183 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:44:38.438099 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:44:38.438056 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:13.803194 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:45:13.803154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:13.899339 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:45:13.899300 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:15.055329 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:45:15.055283 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:15.156111 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:45:15.156058 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:36.590304 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:45:36.590258 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:45:44.399484 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:45:44.399390 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:45:54.193574 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:45:54.193534 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:46:04.591167 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:46:04.591131 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:47:02.812179 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:47:02.812137 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:20.887014 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:47:20.886924 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:22.111547 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:47:22.111515 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:47:43.393315 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:47:43.393276 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:47:55.386071 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:47:55.386038 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:48:01.038037 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:01.037996 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:01.134590 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:01.134550 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:01.275451 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:01.275415 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:06.887971 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:06.887933 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:06.953131 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:06.953095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:07.046766 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:07.046724 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:07.144460 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:07.144370 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:07.267531 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:07.267495 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:13.062138 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:13.062096 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:13.161231 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:13.161187 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:48:31.192524 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:31.192480 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:48:40.991988 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:40.991950 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:48:50.595324 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:50.595235 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:48:58.586406 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:48:58.586371 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:49:10.694394 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:49:10.694354 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:49:22.092487 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:49:22.092448 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:49:33.291195 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:49:33.291157 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:49:43.286831 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:49:43.286788 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:50:05.693044 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:50:05.693001 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:50:08.432468 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:50:08.432434 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:09.971711 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:50:09.971657 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:50:15.087461 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:50:15.087367 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:50:39.193318 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:50:39.193281 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:50:50.480342 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:50:50.480303 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:51:07.588350 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:51:07.588315 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:51:17.797084 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:51:17.797043 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:51:27.888721 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:51:27.888665 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:51:40.391816 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:51:40.391772 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:52:38.257576 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:52:38.257537 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:52:38.345815 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:52:38.345779 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:01.096626 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:53:01.096581 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:21.880347 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:53:21.880309 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:53:35.893838 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:53:35.893805 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:53:48.293601 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:53:48.293562 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:53:57.684239 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:53:57.684202 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:54:07.590231 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:54:07.590191 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:54:18.984693 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:54:18.984637 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:54:27.386709 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:54:27.386657 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:54:37.283309 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:54:37.283271 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:54:48.395360 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:54:48.395285 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:54:56.194043 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:54:56.194013 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:55:07.888091 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:55:07.888048 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:55:45.192846 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:55:45.192808 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:55:55.148056 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:55:55.148012 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:56:03.983597 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:03.983558 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj"] Jul 27 21:56:03.987034 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:03.987015 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:03.989710 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:03.989659 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b1bf115a-kserve-self-signed-certs\"" Jul 27 21:56:04.011659 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.011619 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj"] Jul 27 21:56:04.050952 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.050913 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:04.080301 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.080271 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-model-cache\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.080410 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.080331 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmnmg\" (UniqueName: \"kubernetes.io/projected/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-kube-api-access-zmnmg\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.080410 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.080385 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-dshm\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.080508 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.080434 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tls-certs\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.080508 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.080460 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-home\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.080508 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.080484 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tmp-dir\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182039 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.181995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-dshm\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182039 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182051 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tls-certs\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182326 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182074 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-home\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182326 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182099 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tmp-dir\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182326 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182125 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-model-cache\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182326 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182234 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnmg\" (UniqueName: \"kubernetes.io/projected/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-kube-api-access-zmnmg\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182567 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182532 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-home\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182635 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182584 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-model-cache\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.182635 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.182628 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tmp-dir\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.184614 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.184589 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-dshm\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.184810 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.184793 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tls-certs\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.192744 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.192711 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmnmg\" (UniqueName: \"kubernetes.io/projected/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-kube-api-access-zmnmg\") pod \"e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.295806 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.295716 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:04.453949 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.453908 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj"] Jul 27 21:56:04.457608 ip-10-0-133-37 kubenswrapper[2556]: W0727 21:56:04.457584 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cd6cbf2_87ff_4a29_9bfa_693c988c1230.slice/crio-fab75b864761d23548a9a67dd9ca4aaa33ded750356e4c0a7ad420fb9defc6e4 WatchSource:0}: Error finding container fab75b864761d23548a9a67dd9ca4aaa33ded750356e4c0a7ad420fb9defc6e4: Status 404 returned error can't find the container with id fab75b864761d23548a9a67dd9ca4aaa33ded750356e4c0a7ad420fb9defc6e4 Jul 27 21:56:04.764176 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.764138 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" event={"ID":"0cd6cbf2-87ff-4a29-9bfa-693c988c1230","Type":"ContainerStarted","Data":"6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1"} Jul 27 21:56:04.764392 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.764187 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" event={"ID":"0cd6cbf2-87ff-4a29-9bfa-693c988c1230","Type":"ContainerStarted","Data":"fab75b864761d23548a9a67dd9ca4aaa33ded750356e4c0a7ad420fb9defc6e4"} Jul 27 21:56:04.783735 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:04.783663 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" podStartSLOduration=1.783647916 podStartE2EDuration="1.783647916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:56:04.781736899 +0000 UTC m=+1333.190158930" watchObservedRunningTime="2026-07-27 21:56:04.783647916 +0000 UTC m=+1333.192069923" Jul 27 21:56:07.691473 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:07.691430 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:56:14.296611 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:14.296516 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:14.296611 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:14.296600 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:14.309380 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:14.309349 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:14.827941 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:14.827912 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:56:32.379201 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:32.379166 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:36.007850 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:36.007810 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:56:51.089528 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:51.089492 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj"] Jul 27 21:56:51.118368 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:51.118329 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:52.057146 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:52.057108 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:56:55.814992 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:56:55.814959 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:57:06.109081 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.109018 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" podUID="0cd6cbf2-87ff-4a29-9bfa-693c988c1230" containerName="main" containerID="cri-o://6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1" gracePeriod=45 Jul 27 21:57:06.358469 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.358443 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:57:06.521267 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521220 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tmp-dir\") pod \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " Jul 27 21:57:06.521267 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521275 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-model-cache\") pod \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " Jul 27 21:57:06.521505 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521309 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-home\") pod \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " Jul 27 21:57:06.521505 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521333 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tls-certs\") pod \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " Jul 27 21:57:06.521505 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521359 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmnmg\" (UniqueName: \"kubernetes.io/projected/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-kube-api-access-zmnmg\") pod \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " Jul 27 21:57:06.521505 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521440 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-dshm\") pod \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\" (UID: \"0cd6cbf2-87ff-4a29-9bfa-693c988c1230\") " Jul 27 21:57:06.521712 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521549 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cd6cbf2-87ff-4a29-9bfa-693c988c1230" (UID: "0cd6cbf2-87ff-4a29-9bfa-693c988c1230"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:57:06.521712 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521635 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-home" (OuterVolumeSpecName: "home") pod "0cd6cbf2-87ff-4a29-9bfa-693c988c1230" (UID: "0cd6cbf2-87ff-4a29-9bfa-693c988c1230"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:57:06.521712 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.521653 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cd6cbf2-87ff-4a29-9bfa-693c988c1230" (UID: "0cd6cbf2-87ff-4a29-9bfa-693c988c1230"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:57:06.523818 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.523789 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cd6cbf2-87ff-4a29-9bfa-693c988c1230" (UID: "0cd6cbf2-87ff-4a29-9bfa-693c988c1230"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 21:57:06.523818 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.523804 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-kube-api-access-zmnmg" (OuterVolumeSpecName: "kube-api-access-zmnmg") pod "0cd6cbf2-87ff-4a29-9bfa-693c988c1230" (UID: "0cd6cbf2-87ff-4a29-9bfa-693c988c1230"). InnerVolumeSpecName "kube-api-access-zmnmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 21:57:06.523818 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.523801 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-dshm" (OuterVolumeSpecName: "dshm") pod "0cd6cbf2-87ff-4a29-9bfa-693c988c1230" (UID: "0cd6cbf2-87ff-4a29-9bfa-693c988c1230"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 21:57:06.622316 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.622264 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:57:06.622316 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.622308 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:57:06.622316 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.622317 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:57:06.622316 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.622326 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:57:06.622316 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.622335 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:57:06.622633 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:06.622342 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmnmg\" (UniqueName: \"kubernetes.io/projected/0cd6cbf2-87ff-4a29-9bfa-693c988c1230-kube-api-access-zmnmg\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 21:57:07.100763 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.100721 2556 generic.go:356] "Generic (PLEG): container finished" podID="0cd6cbf2-87ff-4a29-9bfa-693c988c1230" containerID="6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1" exitCode=0 Jul 27 21:57:07.100943 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.100782 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" Jul 27 21:57:07.100943 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.100786 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" event={"ID":"0cd6cbf2-87ff-4a29-9bfa-693c988c1230","Type":"ContainerDied","Data":"6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1"} Jul 27 21:57:07.100943 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.100891 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj" event={"ID":"0cd6cbf2-87ff-4a29-9bfa-693c988c1230","Type":"ContainerDied","Data":"fab75b864761d23548a9a67dd9ca4aaa33ded750356e4c0a7ad420fb9defc6e4"} Jul 27 21:57:07.100943 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.100907 2556 scope.go:117] "RemoveContainer" containerID="6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1" Jul 27 21:57:07.110815 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.110625 2556 scope.go:117] "RemoveContainer" containerID="6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1" Jul 27 21:57:07.111095 ip-10-0-133-37 kubenswrapper[2556]: E0727 21:57:07.110981 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1\": container with ID starting with 6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1 not found: ID does not exist" containerID="6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1" Jul 27 21:57:07.111095 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.111008 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1"} err="failed to get container status \"6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1\": rpc error: code = NotFound desc = could not find container \"6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1\": container with ID starting with 6bff10e945341d28cb41c466f28e43277fea767356f426e2b7cc1ec2b63803a1 not found: ID does not exist" Jul 27 21:57:07.126367 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.126336 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj"] Jul 27 21:57:07.133745 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:07.133719 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b1bf115a-kserve-577d76d8c-qhxbj"] Jul 27 21:57:08.192751 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:08.192706 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd6cbf2-87ff-4a29-9bfa-693c988c1230" path="/var/lib/kubelet/pods/0cd6cbf2-87ff-4a29-9bfa-693c988c1230/volumes" Jul 27 21:57:10.896359 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:10.896318 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:57:19.103069 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:19.103030 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:57:32.506361 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:32.506325 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:57:47.709303 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:47.709221 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:57:57.697858 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:57:57.697818 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:58:08.604600 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:58:08.604562 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:58:22.207873 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:58:22.207836 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:58:41.308732 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:58:41.308697 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:58:59.307835 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:58:59.307794 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:59:07.510839 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:07.510808 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:59:19.800753 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:19.800649 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:59:37.406635 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:37.406596 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:59:49.490445 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:49.490408 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:49.561931 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:49.561889 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:49.799706 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:49.799594 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 21:59:59.069159 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:59.069122 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 21:59:59.993221 ip-10-0-133-37 kubenswrapper[2556]: I0727 21:59:59.993178 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:00:02.128474 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:02.128437 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:15.103077 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:15.103042 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:00:23.964110 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:23.964067 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:27.356276 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:27.356239 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:27.456734 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:27.456696 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:27.827293 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:27.827261 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:00:27.861173 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:27.861139 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:34.429927 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:34.429891 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:34.547898 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:34.547859 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:34.761518 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:34.761483 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:35.093908 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.093819 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6"] Jul 27 22:00:35.094210 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.094197 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd6cbf2-87ff-4a29-9bfa-693c988c1230" containerName="main" Jul 27 22:00:35.094266 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.094212 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd6cbf2-87ff-4a29-9bfa-693c988c1230" containerName="main" Jul 27 22:00:35.094308 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.094279 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd6cbf2-87ff-4a29-9bfa-693c988c1230" containerName="main" Jul 27 22:00:35.097297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.097282 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.100477 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.100458 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b8dde9de-openshift-default-dockercfg-8ljxw\"" Jul 27 22:00:35.118564 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.118535 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6"] Jul 27 22:00:35.139458 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139421 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-podinfo\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139637 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139463 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f218db-8ff5-44dd-8684-9577c53576d1-istiod-ca-cert\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139637 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139545 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-envoy\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139637 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139620 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-certs\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139807 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139692 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-token\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139807 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139760 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-socket\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139807 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139775 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-credential-socket\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139947 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139816 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-data\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.139947 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.139886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-426fr\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-kube-api-access-426fr\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.147906 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.147872 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:35.240615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240579 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-podinfo\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.240615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240617 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f218db-8ff5-44dd-8684-9577c53576d1-istiod-ca-cert\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.240863 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240646 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-envoy\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.240863 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-certs\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.240863 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240742 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-token\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.240863 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240790 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-socket\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.240863 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240812 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-credential-socket\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.240863 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240851 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-data\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.241182 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.240888 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-426fr\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-kube-api-access-426fr\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.241277 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.241248 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-certs\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.241638 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.241293 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-credential-socket\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.241752 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.241284 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-socket\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.241752 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.241448 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f218db-8ff5-44dd-8684-9577c53576d1-istiod-ca-cert\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.241752 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.241476 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-data\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.243097 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.243077 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-envoy\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.243495 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.243462 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-podinfo\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.246661 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.246630 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:35.249709 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.249688 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-token\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.249819 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.249727 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-426fr\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-kube-api-access-426fr\") pod \"e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.408276 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.408184 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:35.561048 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.561010 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6"] Jul 27 22:00:35.564766 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:00:35.564728 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f218db_8ff5_44dd_8684_9577c53576d1.slice/crio-8c6698b2cff30b09ccdbf9246c6db9a17b82de4e4c1754c12865b0e30acc5509 WatchSource:0}: Error finding container 8c6698b2cff30b09ccdbf9246c6db9a17b82de4e4c1754c12865b0e30acc5509: Status 404 returned error can't find the container with id 8c6698b2cff30b09ccdbf9246c6db9a17b82de4e4c1754c12865b0e30acc5509 Jul 27 22:00:35.567538 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.567494 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:35.567661 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.567591 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:35.567661 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:35.567642 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:36.192100 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:36.192063 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" event={"ID":"c6f218db-8ff5-44dd-8684-9577c53576d1","Type":"ContainerStarted","Data":"0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4"} Jul 27 22:00:36.192304 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:36.192115 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" event={"ID":"c6f218db-8ff5-44dd-8684-9577c53576d1","Type":"ContainerStarted","Data":"8c6698b2cff30b09ccdbf9246c6db9a17b82de4e4c1754c12865b0e30acc5509"} Jul 27 22:00:36.218108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:36.218038 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podStartSLOduration=1.21801663 podStartE2EDuration="1.21801663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:36.216274168 +0000 UTC m=+1604.624696211" watchObservedRunningTime="2026-07-27 22:00:36.21801663 +0000 UTC m=+1604.626438662" Jul 27 22:00:36.409092 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:36.409048 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:36.410563 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:36.410532 2556 patch_prober.go:28] interesting pod/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:00:36.410728 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:36.410595 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:00:37.412472 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:37.412441 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:37.625099 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:37.625061 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:37.747656 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:37.747626 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:37.949637 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:37.949600 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:38.111383 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:38.111290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:38.197804 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:38.197773 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:38.198923 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:38.198900 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:38.535542 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:38.535499 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:38.644373 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:38.644340 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:45.301981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:45.301892 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:00:49.174647 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:49.174594 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:00:53.467071 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:53.466637 2556 patch_prober.go:28] interesting pod/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:47020->10.133.0.43:15021: read: connection reset by peer" start-of-body= Jul 27 22:00:53.467071 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:53.466760 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:47020->10.133.0.43:15021: read: connection reset by peer" Jul 27 22:00:54.282868 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:54.282830 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerID="0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4" exitCode=0 Jul 27 22:00:54.283046 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:54.282883 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" event={"ID":"c6f218db-8ff5-44dd-8684-9577c53576d1","Type":"ContainerDied","Data":"0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4"} Jul 27 22:00:54.283307 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:54.283292 2556 scope.go:117] "RemoveContainer" containerID="0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4" Jul 27 22:00:54.284356 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:54.284323 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:54.284482 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:54.284409 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:54.284482 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:54.284454 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:00:55.290261 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:55.290221 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" event={"ID":"c6f218db-8ff5-44dd-8684-9577c53576d1","Type":"ContainerStarted","Data":"a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe"} Jul 27 22:00:55.409097 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:55.409054 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:55.409276 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:55.409256 2556 patch_prober.go:28] interesting pod/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:00:55.409325 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:55.409303 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:00:56.409091 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:56.409054 2556 patch_prober.go:28] interesting pod/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:00:56.409500 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:56.409117 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:00:57.408980 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:57.408938 2556 patch_prober.go:28] interesting pod/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:00:57.409178 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:57.409012 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:00:58.408943 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:58.408903 2556 patch_prober.go:28] interesting pod/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 27 22:00:58.409126 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:58.408968 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 27 22:00:59.412797 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:59.412766 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:59.413183 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:59.413100 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:00:59.413850 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:00:59.413833 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:01:19.135208 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:19.135171 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:19.769540 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:19.769503 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6"] Jul 27 22:01:19.769780 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:19.769726 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" containerID="cri-o://a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe" gracePeriod=30 Jul 27 22:01:20.540527 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:20.540488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:20.660410 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:20.660370 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:21.074100 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:21.074067 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:21.199824 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:21.199777 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:23.022444 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:23.022410 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:01:25.026525 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.026498 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:01:25.094391 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094304 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-token\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094391 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094342 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-podinfo\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094391 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094388 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f218db-8ff5-44dd-8684-9577c53576d1-istiod-ca-cert\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094630 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094559 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-credential-socket\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094630 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094610 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-data\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094752 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094646 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-426fr\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-kube-api-access-426fr\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094752 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094686 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-envoy\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094752 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094726 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-certs\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094905 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094804 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-socket\") pod \"c6f218db-8ff5-44dd-8684-9577c53576d1\" (UID: \"c6f218db-8ff5-44dd-8684-9577c53576d1\") " Jul 27 22:01:25.094905 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094829 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:25.094905 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.094849 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f218db-8ff5-44dd-8684-9577c53576d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:01:25.095118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.095073 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f218db-8ff5-44dd-8684-9577c53576d1-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.095118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.095099 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.095118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.095078 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:25.095452 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.095429 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:25.095813 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.095779 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:25.096884 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.096858 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:01:25.096884 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.096871 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:01:25.097001 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.096904 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-kube-api-access-426fr" (OuterVolumeSpecName: "kube-api-access-426fr") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "kube-api-access-426fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:01:25.097118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.097102 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6f218db-8ff5-44dd-8684-9577c53576d1" (UID: "c6f218db-8ff5-44dd-8684-9577c53576d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:01:25.196403 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.196369 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.196403 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.196395 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.196403 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.196405 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.196403 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.196413 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-426fr\" (UniqueName: \"kubernetes.io/projected/c6f218db-8ff5-44dd-8684-9577c53576d1-kube-api-access-426fr\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.196716 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.196422 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.196716 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.196432 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.196716 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.196440 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f218db-8ff5-44dd-8684-9577c53576d1-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:01:25.445308 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.445275 2556 generic.go:356] "Generic (PLEG): container finished" podID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerID="a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe" exitCode=0 Jul 27 22:01:25.445486 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.445330 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" Jul 27 22:01:25.445486 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.445355 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" event={"ID":"c6f218db-8ff5-44dd-8684-9577c53576d1","Type":"ContainerDied","Data":"a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe"} Jul 27 22:01:25.445486 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.445408 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6" event={"ID":"c6f218db-8ff5-44dd-8684-9577c53576d1","Type":"ContainerDied","Data":"8c6698b2cff30b09ccdbf9246c6db9a17b82de4e4c1754c12865b0e30acc5509"} Jul 27 22:01:25.445486 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.445429 2556 scope.go:117] "RemoveContainer" containerID="a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe" Jul 27 22:01:25.456067 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.456049 2556 scope.go:117] "RemoveContainer" containerID="0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4" Jul 27 22:01:25.464047 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.464027 2556 scope.go:117] "RemoveContainer" containerID="a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe" Jul 27 22:01:25.464296 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:01:25.464276 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe\": container with ID starting with a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe not found: ID does not exist" containerID="a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe" Jul 27 22:01:25.464376 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.464310 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe"} err="failed to get container status \"a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe\": rpc error: code = NotFound desc = could not find container \"a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe\": container with ID starting with a63435581b74401964cbc91d9cb6d3b050d9a9ea1a41da2b381c49996d3d4dbe not found: ID does not exist" Jul 27 22:01:25.464376 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.464335 2556 scope.go:117] "RemoveContainer" containerID="0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4" Jul 27 22:01:25.464583 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:01:25.464565 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4\": container with ID starting with 0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4 not found: ID does not exist" containerID="0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4" Jul 27 22:01:25.464626 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.464590 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4"} err="failed to get container status \"0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4\": rpc error: code = NotFound desc = could not find container \"0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4\": container with ID starting with 0ae16504cb00b3abad55f0bc1150cbd08c39f496d5a5f460a991e1b5e9ff27d4 not found: ID does not exist" Jul 27 22:01:25.472437 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.472413 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6"] Jul 27 22:01:25.482436 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.482400 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b8dde9de-openshift-default-675f7cc6f4-tnln6"] Jul 27 22:01:25.895423 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:25.895341 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:26.111122 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:26.111092 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:26.194309 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:26.194280 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" path="/var/lib/kubelet/pods/c6f218db-8ff5-44dd-8684-9577c53576d1/volumes" Jul 27 22:01:26.496604 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:26.496513 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:27.065145 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:27.065109 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:27.205948 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:27.205914 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:37.531330 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:37.531299 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:01:57.849480 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:57.849452 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:58.173216 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:58.173132 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:58.409053 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:58.409020 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:58.608542 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:58.608507 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:58.751796 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:58.751762 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:58.963884 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:58.963851 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:59.230123 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:59.230037 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:01:59.677560 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:01:59.677482 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:00.154247 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:00.154214 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:02.138967 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:02.138924 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:02.347582 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:02.347548 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:02:02.417423 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:02.417329 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:02.766969 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:02.766939 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:03.098880 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:03.098795 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:03.546531 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:03.546496 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:03.753263 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:03.753233 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:13.132011 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:13.131929 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:02:33.695185 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:33.695151 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:33.802977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:33.802943 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:33.907939 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:33.907906 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:34.022820 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:34.022736 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:35.243072 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:35.243036 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:35.504235 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:35.504128 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:35.761135 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:35.761050 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:36.113346 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.113260 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89"] Jul 27 22:02:36.113913 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.113893 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" Jul 27 22:02:36.113961 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.113917 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" Jul 27 22:02:36.114033 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.114021 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" Jul 27 22:02:36.117485 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.117468 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.132820 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.132797 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-52fa39dc-openshift-default-dockercfg-l4kzv\"" Jul 27 22:02:36.153338 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.153309 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89"] Jul 27 22:02:36.175309 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.175275 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:36.229583 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.229553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jr97\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-kube-api-access-6jr97\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.229848 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.229588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-credential-socket\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.229848 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.229620 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b65d0b-93ae-4676-813a-b3d11694e627-istiod-ca-cert\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.229848 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.229715 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-socket\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.229848 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.229804 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-istio-token\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.230027 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.229870 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-envoy\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.230027 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.229917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b65d0b-93ae-4676-813a-b3d11694e627-istio-podinfo\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.230027 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.230003 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-data\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.230172 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.230027 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-certs\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331030 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.330993 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-istio-token\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331060 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-envoy\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b65d0b-93ae-4676-813a-b3d11694e627-istio-podinfo\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331152 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-data\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331182 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-certs\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331290 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jr97\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-kube-api-access-6jr97\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331320 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-credential-socket\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331372 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b65d0b-93ae-4676-813a-b3d11694e627-istiod-ca-cert\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.331511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331402 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-socket\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.332133 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331586 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-certs\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.332133 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331974 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-data\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.332133 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.331991 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-socket\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.332133 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.332040 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-credential-socket\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.332360 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.332241 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b65d0b-93ae-4676-813a-b3d11694e627-istiod-ca-cert\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.333646 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.333622 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-envoy\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.333857 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.333840 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b65d0b-93ae-4676-813a-b3d11694e627-istio-podinfo\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.343570 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.343544 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-istio-token\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.344454 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.344429 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jr97\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-kube-api-access-6jr97\") pod \"e2e-derive-52fa39dc-openshift-default-675468849b-n6m89\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.427940 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.427849 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:36.500316 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.500278 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:36.600985 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.600962 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89"] Jul 27 22:02:36.603201 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:02:36.603172 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b65d0b_93ae_4676_813a_b3d11694e627.slice/crio-83471f782962b33074616745d8ec443193f0dd093c11ee51431fea49a01c5459 WatchSource:0}: Error finding container 83471f782962b33074616745d8ec443193f0dd093c11ee51431fea49a01c5459: Status 404 returned error can't find the container with id 83471f782962b33074616745d8ec443193f0dd093c11ee51431fea49a01c5459 Jul 27 22:02:36.605376 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.605345 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:36.605433 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.605414 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:36.605470 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.605447 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:36.828106 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.828069 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" event={"ID":"71b65d0b-93ae-4676-813a-b3d11694e627","Type":"ContainerStarted","Data":"8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada"} Jul 27 22:02:36.828314 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.828126 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" event={"ID":"71b65d0b-93ae-4676-813a-b3d11694e627","Type":"ContainerStarted","Data":"83471f782962b33074616745d8ec443193f0dd093c11ee51431fea49a01c5459"} Jul 27 22:02:36.872779 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:36.872722 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" podStartSLOduration=0.872703869 podStartE2EDuration="872.703869ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:02:36.871441439 +0000 UTC m=+1725.279863469" watchObservedRunningTime="2026-07-27 22:02:36.872703869 +0000 UTC m=+1725.281125895" Jul 27 22:02:37.429328 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:37.429294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:37.434390 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:37.434368 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:37.833260 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:37.833225 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:37.834252 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:37.834234 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:37.956242 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:37.956202 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:02:38.385168 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:38.385134 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:38.551381 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:38.551344 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:38.939320 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:38.939281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:39.295521 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:39.295489 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:39.569537 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:39.569452 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:48.771699 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:48.771647 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:02:52.835487 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.835450 2556 patch_prober.go:28] interesting pod/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 27 22:02:52.835926 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.835521 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 27 22:02:52.912556 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.912521 2556 generic.go:356] "Generic (PLEG): container finished" podID="71b65d0b-93ae-4676-813a-b3d11694e627" containerID="8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada" exitCode=0 Jul 27 22:02:52.912729 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.912591 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" event={"ID":"71b65d0b-93ae-4676-813a-b3d11694e627","Type":"ContainerDied","Data":"8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada"} Jul 27 22:02:52.913006 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.912993 2556 scope.go:117] "RemoveContainer" containerID="8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada" Jul 27 22:02:52.913981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.913954 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:52.914080 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.914023 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:52.914080 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:52.914058 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:02:53.923818 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:53.923774 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" event={"ID":"71b65d0b-93ae-4676-813a-b3d11694e627","Type":"ContainerStarted","Data":"d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc"} Jul 27 22:02:54.429100 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:54.429058 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:54.429333 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:54.429205 2556 patch_prober.go:28] interesting pod/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 27 22:02:54.429333 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:54.429271 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 27 22:02:55.428468 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:55.428428 2556 patch_prober.go:28] interesting pod/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 27 22:02:55.429134 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:55.429101 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 27 22:02:56.428599 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:56.428556 2556 patch_prober.go:28] interesting pod/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 27 22:02:56.429523 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:56.429486 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 27 22:02:57.428764 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:57.428726 2556 patch_prober.go:28] interesting pod/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 27 22:02:57.429368 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:57.429339 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 27 22:02:58.432418 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:58.432379 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:58.432925 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:58.432808 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:58.433396 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:58.433378 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:02:59.604452 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:59.604417 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:02:59.866978 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:02:59.866890 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:00.032444 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:00.032409 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:00.212123 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:00.212085 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89"] Jul 27 22:03:00.272205 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:00.272169 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:00.567520 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:00.567431 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:00.968750 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:00.968704 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" containerID="cri-o://d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc" gracePeriod=30 Jul 27 22:03:01.158459 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:01.158420 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:01.374187 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:01.374077 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:03.106984 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:03.106949 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:04.238735 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:04.238699 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:03:04.653923 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:04.653840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:04.791213 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:04.791178 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:06.241817 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.241758 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:03:06.308287 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308258 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b65d0b-93ae-4676-813a-b3d11694e627-istiod-ca-cert\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308458 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308327 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jr97\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-kube-api-access-6jr97\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308458 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308367 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-credential-socket\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308458 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308390 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-istio-token\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308458 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308422 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-data\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308458 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308457 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b65d0b-93ae-4676-813a-b3d11694e627-istio-podinfo\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308703 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308523 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-socket\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308703 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308552 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-envoy\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308703 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308577 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-certs\") pod \"71b65d0b-93ae-4676-813a-b3d11694e627\" (UID: \"71b65d0b-93ae-4676-813a-b3d11694e627\") " Jul 27 22:03:06.308868 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308695 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:03:06.308868 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308755 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b65d0b-93ae-4676-813a-b3d11694e627-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:03:06.308981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308867 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:03:06.308981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308928 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.308981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308947 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.308981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.308962 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b65d0b-93ae-4676-813a-b3d11694e627-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.309184 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.309075 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:03:06.309423 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.309394 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-data" (OuterVolumeSpecName: "istio-data") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:03:06.310728 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.310703 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-kube-api-access-6jr97" (OuterVolumeSpecName: "kube-api-access-6jr97") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "kube-api-access-6jr97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:03:06.310882 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.310863 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:03:06.311403 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.311380 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71b65d0b-93ae-4676-813a-b3d11694e627-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:03:06.311509 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.311399 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-istio-token" (OuterVolumeSpecName: "istio-token") pod "71b65d0b-93ae-4676-813a-b3d11694e627" (UID: "71b65d0b-93ae-4676-813a-b3d11694e627"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:03:06.409615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.409586 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.409615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.409612 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.409615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.409622 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jr97\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-kube-api-access-6jr97\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.409855 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.409631 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b65d0b-93ae-4676-813a-b3d11694e627-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.409855 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.409639 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b65d0b-93ae-4676-813a-b3d11694e627-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:06.409855 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:06.409646 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b65d0b-93ae-4676-813a-b3d11694e627-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:03:07.002733 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.002695 2556 generic.go:356] "Generic (PLEG): container finished" podID="71b65d0b-93ae-4676-813a-b3d11694e627" containerID="d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc" exitCode=0 Jul 27 22:03:07.002934 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.002757 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" Jul 27 22:03:07.002934 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.002781 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" event={"ID":"71b65d0b-93ae-4676-813a-b3d11694e627","Type":"ContainerDied","Data":"d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc"} Jul 27 22:03:07.002934 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.002837 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89" event={"ID":"71b65d0b-93ae-4676-813a-b3d11694e627","Type":"ContainerDied","Data":"83471f782962b33074616745d8ec443193f0dd093c11ee51431fea49a01c5459"} Jul 27 22:03:07.002934 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.002859 2556 scope.go:117] "RemoveContainer" containerID="d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc" Jul 27 22:03:07.012842 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.012814 2556 scope.go:117] "RemoveContainer" containerID="8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada" Jul 27 22:03:07.020825 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.020809 2556 scope.go:117] "RemoveContainer" containerID="d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc" Jul 27 22:03:07.021074 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:03:07.021055 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc\": container with ID starting with d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc not found: ID does not exist" containerID="d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc" Jul 27 22:03:07.021123 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.021081 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc"} err="failed to get container status \"d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc\": rpc error: code = NotFound desc = could not find container \"d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc\": container with ID starting with d17c5b26488038af39fc86a85a9ef0810bca9e5b8850af558cf83423cc0008cc not found: ID does not exist" Jul 27 22:03:07.021123 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.021096 2556 scope.go:117] "RemoveContainer" containerID="8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada" Jul 27 22:03:07.021334 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:03:07.021315 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada\": container with ID starting with 8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada not found: ID does not exist" containerID="8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada" Jul 27 22:03:07.021382 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.021341 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada"} err="failed to get container status \"8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada\": rpc error: code = NotFound desc = could not find container \"8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada\": container with ID starting with 8a8be7b418845f1eb524e9d1fbf882cbc27be742421e4ca9a088976242c05ada not found: ID does not exist" Jul 27 22:03:07.040334 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.040311 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89"] Jul 27 22:03:07.055701 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:07.054075 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-52fa39dc-openshift-default-675468849b-n6m89"] Jul 27 22:03:08.192087 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:08.192052 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" path="/var/lib/kubelet/pods/71b65d0b-93ae-4676-813a-b3d11694e627/volumes" Jul 27 22:03:10.326232 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:10.326192 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:10.514540 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:10.514509 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:10.704162 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:10.704125 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:11.185406 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:11.185371 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:37.154106 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:37.154036 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:44.746713 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:44.746621 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:44.861383 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:44.861354 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:03:44.975226 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:03:44.975195 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:01.007148 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:01.007109 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:07.425591 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:07.425551 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:07.533767 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:07.533730 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:07.641053 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:07.641023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:18.359131 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:18.359094 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:19.186430 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:19.186391 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:21.814019 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:21.813985 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:21.922602 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:21.922569 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:26.923989 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:26.923946 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:27.047679 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:27.047630 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:27.163538 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:27.163498 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:27.257474 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:27.257435 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:28.088481 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:28.088443 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:28.204743 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:28.204704 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:28.297574 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:28.297541 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:29.990375 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:29.990339 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:31.214393 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:31.214350 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:04:37.460846 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:37.460809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:37.571217 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:37.571181 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:38.216519 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:38.216485 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:41.475365 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:41.475326 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:04:44.992039 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:44.992007 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:45.102237 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:45.102202 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:45.232042 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:45.232005 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:45.611249 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:45.611216 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:04:45.823080 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:04:45.823047 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:28.884800 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:28.884698 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:29.015397 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:29.015361 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:29.179910 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:29.179830 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:29.280904 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:29.280871 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:29.413209 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:29.413170 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:29.521497 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:29.521468 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:29.657594 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:29.657557 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:29.768254 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:29.768225 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:30.241772 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:30.241741 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:30.377213 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:30.377178 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:30.951297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:30.951267 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:31.059322 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:31.059290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:31.423418 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:31.423321 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:31.530280 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:31.530247 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:32.039650 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:32.039611 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:32.147335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:32.147297 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:33.749989 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:33.749948 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:33.960595 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:33.960561 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:34.129935 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:34.129846 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:34.240869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:34.240830 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:34.375204 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:34.375164 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:39.950941 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:39.950903 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:40.094467 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:40.094436 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:40.307503 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:40.307419 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:40.493901 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:40.493864 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:40.611156 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:40.611071 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:40.764551 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:40.764512 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:40.895167 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:40.895073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:41.031678 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:41.031635 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:41.155472 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:41.155390 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:41.378865 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:41.378835 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:41.529413 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:41.529376 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:42.468423 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:42.468392 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:42.796919 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:42.796831 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:42.932507 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:42.932474 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:43.093875 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:43.093783 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:43.233554 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:43.233518 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:43.366355 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:43.366264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:49.321639 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:49.321605 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:49.424294 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:49.424262 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:56.199853 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:56.199817 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:56.616339 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:56.616251 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:57.414677 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:57.414640 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:59.872061 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:59.872022 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:05:59.988039 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:05:59.987991 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:00.085871 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:00.085834 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:00.236449 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:00.236416 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:00.710764 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:00.710720 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:00.812940 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:00.812907 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:05.600664 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:05.600625 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:06:24.756811 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:24.756768 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:34.810501 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:34.810460 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:06:38.095277 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:38.095241 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:06:54.795776 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:06:54.795739 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:07:11.759388 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:11.759351 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:11.862543 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:11.862511 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:13.199999 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:13.199963 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:07:14.713087 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:14.713051 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:14.812406 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:14.812369 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:14.885510 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:14.885478 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:18.219489 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:18.219452 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:23.296317 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:23.296281 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:07:23.794894 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:23.794858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:07:59.759723 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:07:59.759688 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:00.307590 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:00.307553 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:05.806687 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:05.806635 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:05.906011 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:05.905976 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:06.135993 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:06.135897 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:06.369678 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:06.369642 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:07.454614 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:07.454579 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:07.544975 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:07.544928 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:07.644433 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:07.644396 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:08.450439 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:08.450391 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:08.541498 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:08.541460 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:09.990029 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:09.989997 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:09.991033 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:09.991010 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:08:15.588845 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:15.588735 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:15.687906 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:15.687854 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:16.256387 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:16.256355 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:22.084727 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:22.084686 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:08:22.302832 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:22.302799 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:22.412393 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:22.412299 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:22.502803 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:22.502770 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:32.163408 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:32.163367 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:33.020023 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:33.019986 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:33.367292 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:33.367212 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:08:54.411891 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:08:54.411854 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:01.169861 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:01.169820 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:01.269920 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:01.269878 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:01.545513 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:01.545476 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:01.866430 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:01.866341 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:01.935903 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:01.935864 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:04.628331 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:04.628290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:04.717475 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:04.717431 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:04.787972 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:04.787936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:04.915419 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:04.915296 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:05.292607 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:05.292574 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:05.381113 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:05.381080 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:10.688689 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:10.688639 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:09:29.755846 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:29.755812 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:30.101108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:30.101011 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:30.434466 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:30.434382 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:30.770991 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:30.770928 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:33.726142 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:33.726102 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:33.806317 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:33.806265 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:33.886291 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:33.886253 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:34.022269 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:34.022170 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:34.422399 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:34.422290 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:34.516822 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:34.516787 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:34.703446 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:34.703414 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:09:41.587753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:41.587717 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:09:58.758004 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:09:58.757945 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:11.084907 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:11.084866 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:11.159321 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:11.159254 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:21.099045 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099010 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2"] Jul 27 22:10:21.099457 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099361 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" Jul 27 22:10:21.099457 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099373 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" Jul 27 22:10:21.099457 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099391 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" Jul 27 22:10:21.099457 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099397 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" Jul 27 22:10:21.099457 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099410 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" Jul 27 22:10:21.099457 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099415 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" Jul 27 22:10:21.099711 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099466 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" Jul 27 22:10:21.099711 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.099480 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f218db-8ff5-44dd-8684-9577c53576d1" containerName="istio-proxy" Jul 27 22:10:21.102619 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.102603 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.110338 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.110311 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-462b96\"/\"kube-root-ca.crt\"" Jul 27 22:10:21.111388 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.111367 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-462b96\"/\"default-dockercfg-fnpzt\"" Jul 27 22:10:21.111491 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.111398 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-462b96\"/\"test-model-462b96-kserve-self-signed-certs\"" Jul 27 22:10:21.111491 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.111433 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-462b96\"/\"openshift-service-ca.crt\"" Jul 27 22:10:21.120566 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.120539 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2"] Jul 27 22:10:21.167011 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.166981 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tmp-dir\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.167011 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.167024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tls-certs\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.167241 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.167055 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-model-cache\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.167241 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.167132 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrlv\" (UniqueName: \"kubernetes.io/projected/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-kube-api-access-sfrlv\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.167241 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.167171 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-home\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.167241 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.167225 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-dshm\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.173053 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.173023 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:21.268058 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268024 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-dshm\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268058 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268067 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tmp-dir\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268322 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268099 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tls-certs\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268322 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268120 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-model-cache\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268322 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268169 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrlv\" (UniqueName: \"kubernetes.io/projected/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-kube-api-access-sfrlv\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268322 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268204 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-home\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268568 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268542 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tmp-dir\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268643 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268591 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-model-cache\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.268643 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.268630 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-home\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.270740 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.270715 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-dshm\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.270981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.270965 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tls-certs\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.277654 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.277628 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrlv\" (UniqueName: \"kubernetes.io/projected/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-kube-api-access-sfrlv\") pod \"test-model-462b96-kserve-647556f76c-dz9z2\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.412391 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.412282 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:21.558420 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:21.558395 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2"] Jul 27 22:10:21.560734 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:10:21.560704 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7120062b_7b85_4c6d_9e53_a00bf3ee8bb8.slice/crio-acab4ac515e0a1dc5af2e76832b6c18255171123c6dbce61d933c7b7fe64a954 WatchSource:0}: Error finding container acab4ac515e0a1dc5af2e76832b6c18255171123c6dbce61d933c7b7fe64a954: Status 404 returned error can't find the container with id acab4ac515e0a1dc5af2e76832b6c18255171123c6dbce61d933c7b7fe64a954 Jul 27 22:10:22.281748 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:22.281710 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" event={"ID":"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8","Type":"ContainerStarted","Data":"431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90"} Jul 27 22:10:22.282196 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:22.281761 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" event={"ID":"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8","Type":"ContainerStarted","Data":"acab4ac515e0a1dc5af2e76832b6c18255171123c6dbce61d933c7b7fe64a954"} Jul 27 22:10:22.303649 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:22.303587 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" podStartSLOduration=1.30356824 podStartE2EDuration="1.30356824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:10:22.300623624 +0000 UTC m=+2190.709045666" watchObservedRunningTime="2026-07-27 22:10:22.30356824 +0000 UTC m=+2190.711990271" Jul 27 22:10:24.390414 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:24.390371 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:10:31.413292 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:31.413193 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:31.413723 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:31.413314 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:31.426306 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:31.426281 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:32.351106 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:32.351073 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:10:37.273729 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.273698 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b"] Jul 27 22:10:37.274314 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.274295 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b65d0b-93ae-4676-813a-b3d11694e627" containerName="istio-proxy" Jul 27 22:10:37.277572 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.277541 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.285397 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.285374 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d28642\"/\"kube-root-ca.crt\"" Jul 27 22:10:37.285814 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.285786 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d28642\"/\"default-dockercfg-p9j9k\"" Jul 27 22:10:37.285895 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.285817 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d28642\"/\"openshift-service-ca.crt\"" Jul 27 22:10:37.288776 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.288753 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d28642\"/\"test-model-d28642-kserve-self-signed-certs\"" Jul 27 22:10:37.291240 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.291210 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b"] Jul 27 22:10:37.306023 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.305990 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cee055-c512-42e8-a060-472a25f6e225-tls-certs\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.306185 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.306047 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-dshm\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.306185 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.306097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5lp\" (UniqueName: \"kubernetes.io/projected/81cee055-c512-42e8-a060-472a25f6e225-kube-api-access-bj5lp\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.306280 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.306192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-tmp-dir\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.306280 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.306232 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-model-cache\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.306280 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.306258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-home\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.314857 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.314822 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:10:37.406719 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.406646 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cee055-c512-42e8-a060-472a25f6e225-tls-certs\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.406891 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.406746 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-dshm\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.406891 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.406800 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5lp\" (UniqueName: \"kubernetes.io/projected/81cee055-c512-42e8-a060-472a25f6e225-kube-api-access-bj5lp\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.406891 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.406869 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-tmp-dir\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.407048 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.406902 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-model-cache\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.407048 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.406930 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-home\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.407328 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.407306 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-tmp-dir\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.407515 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.407316 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-model-cache\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.407515 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.407342 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-home\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.409176 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.409154 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-dshm\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.409408 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.409391 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cee055-c512-42e8-a060-472a25f6e225-tls-certs\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.417582 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.417556 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5lp\" (UniqueName: \"kubernetes.io/projected/81cee055-c512-42e8-a060-472a25f6e225-kube-api-access-bj5lp\") pod \"test-model-d28642-kserve-869d96488f-jkd2b\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.587896 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.587808 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:37.730439 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:37.730412 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b"] Jul 27 22:10:37.732394 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:10:37.732363 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81cee055_c512_42e8_a060_472a25f6e225.slice/crio-9f4a1c81b19c516445f571db9a07b7a087f34355ab817423b3ad6258ac726003 WatchSource:0}: Error finding container 9f4a1c81b19c516445f571db9a07b7a087f34355ab817423b3ad6258ac726003: Status 404 returned error can't find the container with id 9f4a1c81b19c516445f571db9a07b7a087f34355ab817423b3ad6258ac726003 Jul 27 22:10:38.375865 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:38.375826 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" event={"ID":"81cee055-c512-42e8-a060-472a25f6e225","Type":"ContainerStarted","Data":"c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e"} Jul 27 22:10:38.375865 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:38.375872 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" event={"ID":"81cee055-c512-42e8-a060-472a25f6e225","Type":"ContainerStarted","Data":"9f4a1c81b19c516445f571db9a07b7a087f34355ab817423b3ad6258ac726003"} Jul 27 22:10:38.395148 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:38.395102 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" podStartSLOduration=1.395089126 podStartE2EDuration="1.395089126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:10:38.393458161 +0000 UTC m=+2206.801880192" watchObservedRunningTime="2026-07-27 22:10:38.395089126 +0000 UTC m=+2206.803511197" Jul 27 22:10:38.595057 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:38.595017 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:10:47.588190 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:47.588146 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:47.588190 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:47.588203 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:47.601043 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:47.601007 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:48.440648 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:48.440622 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:10:56.120052 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:10:56.120015 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:11:23.697115 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:23.697017 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2"] Jul 27 22:11:23.720734 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:23.720697 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:24.611696 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:24.611643 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b"] Jul 27 22:11:24.631203 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:24.631165 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:28.707685 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:28.707641 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:11:34.091707 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:34.091619 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:35.144581 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:35.144544 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:35.221351 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:35.221300 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:35.298567 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:35.298510 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:35.366152 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:35.366099 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:35.753436 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:35.753385 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:38.719459 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:38.719414 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" podUID="7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" containerName="main" containerID="cri-o://431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90" gracePeriod=45 Jul 27 22:11:38.974883 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:38.974812 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:11:39.072358 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072322 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-home\") pod \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " Jul 27 22:11:39.072358 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072364 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-model-cache\") pod \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " Jul 27 22:11:39.072646 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072392 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfrlv\" (UniqueName: \"kubernetes.io/projected/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-kube-api-access-sfrlv\") pod \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " Jul 27 22:11:39.072646 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072421 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tmp-dir\") pod \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " Jul 27 22:11:39.072646 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072480 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tls-certs\") pod \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " Jul 27 22:11:39.072646 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072497 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-dshm\") pod \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\" (UID: \"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8\") " Jul 27 22:11:39.072942 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072757 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-home" (OuterVolumeSpecName: "home") pod "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" (UID: "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.072942 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072770 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" (UID: "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.072942 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.072784 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" (UID: "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.075007 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.074972 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-dshm" (OuterVolumeSpecName: "dshm") pod "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" (UID: "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.075130 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.075017 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" (UID: "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:11:39.075130 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.075034 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-kube-api-access-sfrlv" (OuterVolumeSpecName: "kube-api-access-sfrlv") pod "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" (UID: "7120062b-7b85-4c6d-9e53-a00bf3ee8bb8"). InnerVolumeSpecName "kube-api-access-sfrlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:11:39.173869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.173828 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.173869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.173863 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.173869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.173872 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.173869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.173881 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.174147 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.173889 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfrlv\" (UniqueName: \"kubernetes.io/projected/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-kube-api-access-sfrlv\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.174147 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.173920 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.631771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.631726 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" podUID="81cee055-c512-42e8-a060-472a25f6e225" containerName="main" containerID="cri-o://c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e" gracePeriod=45 Jul 27 22:11:39.694415 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.694380 2556 generic.go:356] "Generic (PLEG): container finished" podID="7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" containerID="431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90" exitCode=0 Jul 27 22:11:39.694576 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.694433 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" Jul 27 22:11:39.694576 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.694468 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" event={"ID":"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8","Type":"ContainerDied","Data":"431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90"} Jul 27 22:11:39.694576 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.694519 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2" event={"ID":"7120062b-7b85-4c6d-9e53-a00bf3ee8bb8","Type":"ContainerDied","Data":"acab4ac515e0a1dc5af2e76832b6c18255171123c6dbce61d933c7b7fe64a954"} Jul 27 22:11:39.694576 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.694540 2556 scope.go:117] "RemoveContainer" containerID="431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90" Jul 27 22:11:39.715355 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.715300 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2"] Jul 27 22:11:39.718123 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.718088 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-462b96/test-model-462b96-kserve-647556f76c-dz9z2"] Jul 27 22:11:39.742616 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.742485 2556 scope.go:117] "RemoveContainer" containerID="431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90" Jul 27 22:11:39.742925 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:11:39.742891 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90\": container with ID starting with 431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90 not found: ID does not exist" containerID="431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90" Jul 27 22:11:39.742991 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.742919 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90"} err="failed to get container status \"431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90\": rpc error: code = NotFound desc = could not find container \"431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90\": container with ID starting with 431287e62fe142b25a403defb47457109a898f266ae08e8aec0a1f3e35317e90 not found: ID does not exist" Jul 27 22:11:39.892460 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.892395 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:11:39.982587 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982554 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-tmp-dir\") pod \"81cee055-c512-42e8-a060-472a25f6e225\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " Jul 27 22:11:39.982771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982623 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj5lp\" (UniqueName: \"kubernetes.io/projected/81cee055-c512-42e8-a060-472a25f6e225-kube-api-access-bj5lp\") pod \"81cee055-c512-42e8-a060-472a25f6e225\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " Jul 27 22:11:39.982771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982700 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-model-cache\") pod \"81cee055-c512-42e8-a060-472a25f6e225\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " Jul 27 22:11:39.982771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982716 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-home\") pod \"81cee055-c512-42e8-a060-472a25f6e225\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " Jul 27 22:11:39.982771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982756 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cee055-c512-42e8-a060-472a25f6e225-tls-certs\") pod \"81cee055-c512-42e8-a060-472a25f6e225\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " Jul 27 22:11:39.982771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982770 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-dshm\") pod \"81cee055-c512-42e8-a060-472a25f6e225\" (UID: \"81cee055-c512-42e8-a060-472a25f6e225\") " Jul 27 22:11:39.983010 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982900 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81cee055-c512-42e8-a060-472a25f6e225" (UID: "81cee055-c512-42e8-a060-472a25f6e225"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.983010 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.982984 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-model-cache" (OuterVolumeSpecName: "model-cache") pod "81cee055-c512-42e8-a060-472a25f6e225" (UID: "81cee055-c512-42e8-a060-472a25f6e225"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.983130 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.983021 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-home" (OuterVolumeSpecName: "home") pod "81cee055-c512-42e8-a060-472a25f6e225" (UID: "81cee055-c512-42e8-a060-472a25f6e225"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.983130 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.983052 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.983130 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.983076 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:39.985073 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.985046 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-dshm" (OuterVolumeSpecName: "dshm") pod "81cee055-c512-42e8-a060-472a25f6e225" (UID: "81cee055-c512-42e8-a060-472a25f6e225"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:11:39.985073 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.985051 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cee055-c512-42e8-a060-472a25f6e225-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81cee055-c512-42e8-a060-472a25f6e225" (UID: "81cee055-c512-42e8-a060-472a25f6e225"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:11:39.985292 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:39.985084 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cee055-c512-42e8-a060-472a25f6e225-kube-api-access-bj5lp" (OuterVolumeSpecName: "kube-api-access-bj5lp") pod "81cee055-c512-42e8-a060-472a25f6e225" (UID: "81cee055-c512-42e8-a060-472a25f6e225"). InnerVolumeSpecName "kube-api-access-bj5lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:11:40.084105 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.084069 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5lp\" (UniqueName: \"kubernetes.io/projected/81cee055-c512-42e8-a060-472a25f6e225-kube-api-access-bj5lp\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:40.084105 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.084099 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:40.084105 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.084108 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81cee055-c512-42e8-a060-472a25f6e225-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:40.084355 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.084118 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81cee055-c512-42e8-a060-472a25f6e225-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:11:40.193360 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.193273 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" path="/var/lib/kubelet/pods/7120062b-7b85-4c6d-9e53-a00bf3ee8bb8/volumes" Jul 27 22:11:40.704302 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.704261 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" Jul 27 22:11:40.704560 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.704267 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" event={"ID":"81cee055-c512-42e8-a060-472a25f6e225","Type":"ContainerDied","Data":"c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e"} Jul 27 22:11:40.704560 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.704186 2556 generic.go:356] "Generic (PLEG): container finished" podID="81cee055-c512-42e8-a060-472a25f6e225" containerID="c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e" exitCode=0 Jul 27 22:11:40.704560 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.704445 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b" event={"ID":"81cee055-c512-42e8-a060-472a25f6e225","Type":"ContainerDied","Data":"9f4a1c81b19c516445f571db9a07b7a087f34355ab817423b3ad6258ac726003"} Jul 27 22:11:40.704560 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.704480 2556 scope.go:117] "RemoveContainer" containerID="c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e" Jul 27 22:11:40.713823 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.713804 2556 scope.go:117] "RemoveContainer" containerID="c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e" Jul 27 22:11:40.714110 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:11:40.714089 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e\": container with ID starting with c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e not found: ID does not exist" containerID="c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e" Jul 27 22:11:40.714196 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.714126 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e"} err="failed to get container status \"c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e\": rpc error: code = NotFound desc = could not find container \"c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e\": container with ID starting with c8d38f6514a8cbc0313853552915fa3876b6bb2daa3f618ba1e8280dce5e304e not found: ID does not exist" Jul 27 22:11:40.721763 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.721738 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b"] Jul 27 22:11:40.725497 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:40.725475 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d28642/test-model-d28642-kserve-869d96488f-jkd2b"] Jul 27 22:11:41.425955 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:41.425925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:41.525714 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:41.525650 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:41.694089 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:41.694059 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:41.927959 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:41.927925 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:42.192514 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:42.192433 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81cee055-c512-42e8-a060-472a25f6e225" path="/var/lib/kubelet/pods/81cee055-c512-42e8-a060-472a25f6e225/volumes" Jul 27 22:11:44.789118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:44.789082 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:11:52.453075 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:52.453006 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:52.521743 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:52.521706 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:52.622182 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:52.622092 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:52.697361 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:52.697330 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:53.014427 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:53.014380 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:56.791869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:56.791833 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:11:59.016416 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:59.016377 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:59.117255 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:59.117220 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:59.276091 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:59.276001 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:59.517006 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:59.516966 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:11:59.864339 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:11:59.864303 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:00.210197 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:00.210154 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:00.847731 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:00.847697 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:03.486838 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:03.486797 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:03.571816 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:03.571773 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:03.641095 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:03.641060 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:04.072791 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:04.072756 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:04.174757 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:04.174721 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:08.592370 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:08.592335 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:12:28.522848 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:28.522816 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:29.010859 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.010821 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:29.299274 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.299179 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t"] Jul 27 22:12:29.299742 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.299719 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81cee055-c512-42e8-a060-472a25f6e225" containerName="main" Jul 27 22:12:29.299742 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.299744 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81cee055-c512-42e8-a060-472a25f6e225" containerName="main" Jul 27 22:12:29.299902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.299769 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" containerName="main" Jul 27 22:12:29.299902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.299780 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" containerName="main" Jul 27 22:12:29.299902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.299874 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7120062b-7b85-4c6d-9e53-a00bf3ee8bb8" containerName="main" Jul 27 22:12:29.299902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.299886 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="81cee055-c512-42e8-a060-472a25f6e225" containerName="main" Jul 27 22:12:29.302990 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.302969 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.305538 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.305519 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5e1a2f-openshift-default-dockercfg-qf9d7\"" Jul 27 22:12:29.316585 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.316556 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t"] Jul 27 22:12:29.327695 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.327632 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:29.402375 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402340 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-credential-socket\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402375 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402373 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-data\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402663 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402419 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-token\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402663 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402440 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-socket\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402663 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402456 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-podinfo\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402663 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402485 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxt8x\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-kube-api-access-fxt8x\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402663 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402548 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4471896b-3aed-460a-9a4a-3cd2678eb86e-istiod-ca-cert\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402663 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402624 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-envoy\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.402936 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.402747 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-certs\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.403737 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.403650 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:29.503865 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.503827 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-envoy\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.503878 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-certs\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.503899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-credential-socket\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.503917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-data\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.503961 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-token\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.503993 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-socket\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504019 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-podinfo\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504050 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxt8x\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-kube-api-access-fxt8x\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504397 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504075 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4471896b-3aed-460a-9a4a-3cd2678eb86e-istiod-ca-cert\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504397 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504290 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-certs\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504397 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504380 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-credential-socket\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504527 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504448 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-data\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504642 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504625 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-socket\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.504774 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.504757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4471896b-3aed-460a-9a4a-3cd2678eb86e-istiod-ca-cert\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.506332 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.506317 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-envoy\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.506764 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.506746 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-podinfo\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.517059 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.517025 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxt8x\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-kube-api-access-fxt8x\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.517346 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.517329 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-token\") pod \"e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.614476 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.614396 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:29.762755 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.762729 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t"] Jul 27 22:12:29.764278 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:12:29.764248 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4471896b_3aed_460a_9a4a_3cd2678eb86e.slice/crio-93c22c3391af8eaf52dbc4cb2953e8d1ea6694abb6198134a1938a31a00d234a WatchSource:0}: Error finding container 93c22c3391af8eaf52dbc4cb2953e8d1ea6694abb6198134a1938a31a00d234a: Status 404 returned error can't find the container with id 93c22c3391af8eaf52dbc4cb2953e8d1ea6694abb6198134a1938a31a00d234a Jul 27 22:12:29.766344 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.766310 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:12:29.766439 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.766373 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:12:29.766439 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.766401 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:12:29.964528 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.964481 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" event={"ID":"4471896b-3aed-460a-9a4a-3cd2678eb86e","Type":"ContainerStarted","Data":"77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b"} Jul 27 22:12:29.964757 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.964593 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" event={"ID":"4471896b-3aed-460a-9a4a-3cd2678eb86e","Type":"ContainerStarted","Data":"93c22c3391af8eaf52dbc4cb2953e8d1ea6694abb6198134a1938a31a00d234a"} Jul 27 22:12:29.983289 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:29.983234 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" podStartSLOduration=0.983212707 podStartE2EDuration="983.212707ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:29.982605616 +0000 UTC m=+2318.391027645" watchObservedRunningTime="2026-07-27 22:12:29.983212707 +0000 UTC m=+2318.391634732" Jul 27 22:12:30.615125 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:30.615091 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:30.620309 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:30.620282 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:30.969546 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:30.969521 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:30.970452 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:30.970433 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:31.915131 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:31.915091 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:32.008204 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:32.008167 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:32.107176 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:32.107082 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:32.301386 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:32.301343 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:32.573016 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:32.572932 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:32.657849 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:32.657818 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:12:40.887946 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:40.887908 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:12:49.080178 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:49.080095 2556 generic.go:356] "Generic (PLEG): container finished" podID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerID="77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b" exitCode=0 Jul 27 22:12:49.080178 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:49.080152 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" event={"ID":"4471896b-3aed-460a-9a4a-3cd2678eb86e","Type":"ContainerDied","Data":"77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b"} Jul 27 22:12:49.080652 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:49.080521 2556 scope.go:117] "RemoveContainer" containerID="77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b" Jul 27 22:12:49.081607 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:49.081572 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:12:49.081723 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:49.081658 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:12:49.081793 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:49.081741 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:12:50.088434 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:50.088401 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" event={"ID":"4471896b-3aed-460a-9a4a-3cd2678eb86e","Type":"ContainerStarted","Data":"38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c"} Jul 27 22:12:50.615499 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:50.615460 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:50.615728 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:50.615586 2556 patch_prober.go:28] interesting pod/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:12:50.615728 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:50.615630 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:12:51.615334 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:51.615296 2556 patch_prober.go:28] interesting pod/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:12:51.615953 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:51.615917 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:12:52.615324 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:52.615284 2556 patch_prober.go:28] interesting pod/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:12:52.615619 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:52.615587 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:12:53.614902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:53.614867 2556 patch_prober.go:28] interesting pod/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 27 22:12:53.615224 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:53.615194 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 27 22:12:54.619169 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:54.619137 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:54.619645 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:54.619424 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:54.620136 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:54.620120 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:12:56.938743 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:12:56.938708 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:15.416507 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:15.416473 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:15.521313 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:15.521279 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:15.638913 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:15.638876 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:16.026776 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:16.026740 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:21.887437 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:21.887400 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:21.988313 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:21.988282 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:22.184335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:22.184246 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:22.227634 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:22.227597 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:13:22.456296 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:22.456260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:31.263396 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:31.263357 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:31.340885 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:31.340851 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:31.432297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:31.432259 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:31.526625 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:31.526535 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:31.841737 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:31.841629 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:35.804787 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:35.804749 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:13:38.150361 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:38.150323 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:38.253246 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:38.253206 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:38.411233 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:38.411130 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t"] Jul 27 22:13:38.411462 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:38.411428 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" containerID="cri-o://38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c" gracePeriod=30 Jul 27 22:13:38.440430 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:38.440397 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:38.679634 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:38.679546 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:39.011038 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:39.011008 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:39.371717 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:39.371605 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:39.621936 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:39.621839 2556 patch_prober.go:28] interesting pod/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 22:13:39.621936 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:39.621897 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 22:13:39.726869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:39.726834 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:42.116999 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:42.116944 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:42.220051 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:42.220004 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:42.307031 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:42.306990 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:42.612268 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:42.612231 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:42.921366 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:42.921282 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:43.028530 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.028487 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:13:43.655514 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.655490 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:13:43.793597 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793559 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxt8x\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-kube-api-access-fxt8x\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.793597 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793601 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-certs\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.793917 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793624 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-token\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.793917 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793664 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-credential-socket\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.793917 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793725 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-socket\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.793917 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793767 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-podinfo\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.793917 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793850 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-envoy\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.793917 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793918 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-data\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.794333 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793941 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4471896b-3aed-460a-9a4a-3cd2678eb86e-istiod-ca-cert\") pod \"4471896b-3aed-460a-9a4a-3cd2678eb86e\" (UID: \"4471896b-3aed-460a-9a4a-3cd2678eb86e\") " Jul 27 22:13:43.794333 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793972 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:43.794333 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.793995 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:43.794333 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.794300 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.794333 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.794321 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.794796 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.794339 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4471896b-3aed-460a-9a4a-3cd2678eb86e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:13:43.794796 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.794149 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:43.795003 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.794976 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:43.796163 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.796123 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:13:43.796290 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.796197 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:13:43.796290 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.796216 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-kube-api-access-fxt8x" (OuterVolumeSpecName: "kube-api-access-fxt8x") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "kube-api-access-fxt8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:13:43.796290 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.796268 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4471896b-3aed-460a-9a4a-3cd2678eb86e" (UID: "4471896b-3aed-460a-9a4a-3cd2678eb86e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:13:43.894916 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.894863 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.894916 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.894908 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.895137 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.894924 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4471896b-3aed-460a-9a4a-3cd2678eb86e-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.895137 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.894949 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxt8x\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-kube-api-access-fxt8x\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.895137 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.894960 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.895137 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.894968 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4471896b-3aed-460a-9a4a-3cd2678eb86e-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:43.895137 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:43.894976 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4471896b-3aed-460a-9a4a-3cd2678eb86e-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:13:44.387132 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.387099 2556 generic.go:356] "Generic (PLEG): container finished" podID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerID="38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c" exitCode=0 Jul 27 22:13:44.387299 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.387154 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" Jul 27 22:13:44.387299 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.387169 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" event={"ID":"4471896b-3aed-460a-9a4a-3cd2678eb86e","Type":"ContainerDied","Data":"38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c"} Jul 27 22:13:44.387299 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.387219 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t" event={"ID":"4471896b-3aed-460a-9a4a-3cd2678eb86e","Type":"ContainerDied","Data":"93c22c3391af8eaf52dbc4cb2953e8d1ea6694abb6198134a1938a31a00d234a"} Jul 27 22:13:44.387299 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.387243 2556 scope.go:117] "RemoveContainer" containerID="38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c" Jul 27 22:13:44.396282 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.396262 2556 scope.go:117] "RemoveContainer" containerID="77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b" Jul 27 22:13:44.404378 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.404360 2556 scope.go:117] "RemoveContainer" containerID="38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c" Jul 27 22:13:44.404653 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:13:44.404634 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c\": container with ID starting with 38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c not found: ID does not exist" containerID="38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c" Jul 27 22:13:44.404734 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.404662 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c"} err="failed to get container status \"38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c\": rpc error: code = NotFound desc = could not find container \"38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c\": container with ID starting with 38a466e9677793546a2fa87cb1f0b193d7dc767ba65fdd6ebc12da2f103c468c not found: ID does not exist" Jul 27 22:13:44.404734 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.404700 2556 scope.go:117] "RemoveContainer" containerID="77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b" Jul 27 22:13:44.404948 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:13:44.404934 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b\": container with ID starting with 77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b not found: ID does not exist" containerID="77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b" Jul 27 22:13:44.404989 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.404952 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b"} err="failed to get container status \"77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b\": rpc error: code = NotFound desc = could not find container \"77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b\": container with ID starting with 77ba87cd494eaebdbbada817629748003bc787f181f8cbf2b113c6322a95746b not found: ID does not exist" Jul 27 22:13:44.414633 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.414609 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t"] Jul 27 22:13:44.420386 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:44.420360 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5e1a2f-openshift-default-85fc78895d-zsq5t"] Jul 27 22:13:46.192683 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:46.192632 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" path="/var/lib/kubelet/pods/4471896b-3aed-460a-9a4a-3cd2678eb86e/volumes" Jul 27 22:13:47.025091 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:13:47.025055 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:14:07.115487 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:07.115439 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:07.486276 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:07.486240 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:07.938308 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:07.938218 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:08.152845 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:08.152809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:11.324504 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:11.324469 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:11.538540 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:11.538508 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:12.448362 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:12.448274 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:12.571080 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:12.571044 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:13.404645 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:13.404609 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:13.520481 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:13.520447 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:18.498448 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:18.498406 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:14:26.057119 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:26.057080 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:32.491017 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.490975 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs"] Jul 27 22:14:32.491530 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.491509 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" Jul 27 22:14:32.491615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.491535 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" Jul 27 22:14:32.491615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.491552 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" Jul 27 22:14:32.491615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.491560 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" Jul 27 22:14:32.491819 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.491696 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" Jul 27 22:14:32.494737 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.494715 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.502958 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.502934 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"auth-test-model-94450e-kserve-self-signed-certs\"" Jul 27 22:14:32.503102 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.502941 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"openshift-service-ca.crt\"" Jul 27 22:14:32.503102 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.502941 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"kube-root-ca.crt\"" Jul 27 22:14:32.503102 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.502963 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"default-dockercfg-f9zxb\"" Jul 27 22:14:32.513813 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.513787 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs"] Jul 27 22:14:32.623354 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.623324 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-dshm\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.623534 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.623365 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tls-certs\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.623534 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.623400 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tmp-dir\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.623534 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.623465 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-home\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.623534 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.623496 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4v4\" (UniqueName: \"kubernetes.io/projected/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-kube-api-access-wh4v4\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.623751 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.623535 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-model-cache\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.697297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.697223 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:14:32.723988 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.723957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-dshm\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.723988 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.723994 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tls-certs\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.724256 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.724023 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tmp-dir\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.724256 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.724044 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-home\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.724256 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.724063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4v4\" (UniqueName: \"kubernetes.io/projected/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-kube-api-access-wh4v4\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.724256 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.724136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-model-cache\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.724527 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.724504 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-home\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.724657 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.724557 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tmp-dir\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.724657 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.724596 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-model-cache\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.726694 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.726649 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-dshm\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.727041 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.727020 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tls-certs\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.741185 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.741127 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4v4\" (UniqueName: \"kubernetes.io/projected/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-kube-api-access-wh4v4\") pod \"auth-test-model-94450e-kserve-5b48589bd7-k6dxs\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.804860 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.804802 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:32.952422 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:32.952364 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs"] Jul 27 22:14:32.954777 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:14:32.954739 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c75e242_15d4_4f8f_81ed_b5c07d5b20a7.slice/crio-01f4f1d4748f71316b5eaf182cbed1bfaf3107983142399468af4e3a279820da WatchSource:0}: Error finding container 01f4f1d4748f71316b5eaf182cbed1bfaf3107983142399468af4e3a279820da: Status 404 returned error can't find the container with id 01f4f1d4748f71316b5eaf182cbed1bfaf3107983142399468af4e3a279820da Jul 27 22:14:33.652462 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:33.652426 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" event={"ID":"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7","Type":"ContainerStarted","Data":"587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782"} Jul 27 22:14:33.652462 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:33.652466 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" event={"ID":"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7","Type":"ContainerStarted","Data":"01f4f1d4748f71316b5eaf182cbed1bfaf3107983142399468af4e3a279820da"} Jul 27 22:14:33.672487 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:33.672409 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" podStartSLOduration=1.672393002 podStartE2EDuration="1.672393002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:14:33.671493625 +0000 UTC m=+2442.079915667" watchObservedRunningTime="2026-07-27 22:14:33.672393002 +0000 UTC m=+2442.080815031" Jul 27 22:14:35.805382 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:35.805339 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:14:42.805358 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:42.805320 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:42.805869 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:42.805378 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:42.821239 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:42.821213 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:43.720947 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:43.720913 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:14:58.903653 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:14:58.903609 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:02.795634 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:15:02.795600 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:15:03.409465 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:15:03.409427 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:03.571116 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:15:03.571073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:32.792733 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:15:32.792685 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:15:35.145152 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:15:35.145095 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:15:54.505724 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:15:54.505685 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:16:13.122907 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:16:13.122858 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:16:40.095442 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:16:40.095405 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:16:57.891932 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:16:57.891892 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:17:07.841094 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:17:07.841060 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:09.109408 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:17:09.109378 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:20.596595 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:17:20.596514 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:17:30.896094 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:17:30.896044 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:31.000147 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:17:31.000113 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:37.955909 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:17:37.955877 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:17:38.897991 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:17:38.897956 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:18:05.799824 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:18:05.799775 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:18:24.286488 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:18:24.286452 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:18:50.987562 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:18:50.987527 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:19:09.295518 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:09.295476 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:19:16.605529 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:16.605494 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:36.285789 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:36.285744 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:19:39.597208 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:39.597178 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs"] Jul 27 22:19:39.639984 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:39.639938 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:40.396406 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:40.396367 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:48.291252 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.291207 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn"] Jul 27 22:19:48.291735 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.291616 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4471896b-3aed-460a-9a4a-3cd2678eb86e" containerName="istio-proxy" Jul 27 22:19:48.295019 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.294999 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.298507 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.298488 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"sub-test-model-94450e-kserve-self-signed-certs\"" Jul 27 22:19:48.312549 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.312523 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn"] Jul 27 22:19:48.335034 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.334993 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:19:48.411180 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.411136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-dshm\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.411180 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.411180 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69274bfc-81d9-44a0-aaf4-1301c8070f75-tls-certs\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.411449 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.411205 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-home\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.411449 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.411283 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4cv\" (UniqueName: \"kubernetes.io/projected/69274bfc-81d9-44a0-aaf4-1301c8070f75-kube-api-access-9f4cv\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.411449 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.411326 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-model-cache\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.411449 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.411415 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-tmp-dir\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.512610 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.512563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-tmp-dir\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.512610 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.512608 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-dshm\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.512874 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.512633 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69274bfc-81d9-44a0-aaf4-1301c8070f75-tls-certs\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.512874 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.512655 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-home\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.512874 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.512714 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4cv\" (UniqueName: \"kubernetes.io/projected/69274bfc-81d9-44a0-aaf4-1301c8070f75-kube-api-access-9f4cv\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.512874 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.512737 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-model-cache\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.513155 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.513127 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-home\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.513254 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.513189 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-model-cache\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.513254 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.513235 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-tmp-dir\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.515098 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.515080 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-dshm\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.515522 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.515491 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69274bfc-81d9-44a0-aaf4-1301c8070f75-tls-certs\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.523345 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.523309 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4cv\" (UniqueName: \"kubernetes.io/projected/69274bfc-81d9-44a0-aaf4-1301c8070f75-kube-api-access-9f4cv\") pod \"sub-test-model-94450e-kserve-7578bc7fb5-h28dn\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.604331 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.604233 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:48.753336 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:48.753310 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn"] Jul 27 22:19:48.755368 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:19:48.755343 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69274bfc_81d9_44a0_aaf4_1301c8070f75.slice/crio-fa140114532e3c1dcd1bbfd8dadf0e2eb9f284ed3d3c23e8ff9e236a68400404 WatchSource:0}: Error finding container fa140114532e3c1dcd1bbfd8dadf0e2eb9f284ed3d3c23e8ff9e236a68400404: Status 404 returned error can't find the container with id fa140114532e3c1dcd1bbfd8dadf0e2eb9f284ed3d3c23e8ff9e236a68400404 Jul 27 22:19:49.314873 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:49.314836 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" event={"ID":"69274bfc-81d9-44a0-aaf4-1301c8070f75","Type":"ContainerStarted","Data":"ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695"} Jul 27 22:19:49.315276 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:49.314885 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" event={"ID":"69274bfc-81d9-44a0-aaf4-1301c8070f75","Type":"ContainerStarted","Data":"fa140114532e3c1dcd1bbfd8dadf0e2eb9f284ed3d3c23e8ff9e236a68400404"} Jul 27 22:19:49.333026 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:49.332977 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" podStartSLOduration=1.332962366 podStartE2EDuration="1.332962366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:19:49.331573855 +0000 UTC m=+2757.739995886" watchObservedRunningTime="2026-07-27 22:19:49.332962366 +0000 UTC m=+2757.741384397" Jul 27 22:19:52.134266 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:52.134224 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:19:54.622309 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.622260 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" podUID="4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" containerName="main" containerID="cri-o://587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782" gracePeriod=45 Jul 27 22:19:54.881654 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.881588 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:19:54.974033 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.973999 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-model-cache\") pod \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " Jul 27 22:19:54.974033 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974040 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-dshm\") pod \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " Jul 27 22:19:54.974297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974069 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tls-certs\") pod \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " Jul 27 22:19:54.974297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974144 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tmp-dir\") pod \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " Jul 27 22:19:54.974297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974254 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-home\") pod \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " Jul 27 22:19:54.974297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974284 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh4v4\" (UniqueName: \"kubernetes.io/projected/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-kube-api-access-wh4v4\") pod \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\" (UID: \"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7\") " Jul 27 22:19:54.974559 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974342 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" (UID: "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:54.974559 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974393 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" (UID: "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:54.974559 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974518 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-home" (OuterVolumeSpecName: "home") pod "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" (UID: "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:54.974736 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974622 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:19:54.974736 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974641 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:19:54.974736 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.974655 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:19:54.976713 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.976686 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-kube-api-access-wh4v4" (OuterVolumeSpecName: "kube-api-access-wh4v4") pod "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" (UID: "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7"). InnerVolumeSpecName "kube-api-access-wh4v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:19:54.976938 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.976912 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-dshm" (OuterVolumeSpecName: "dshm") pod "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" (UID: "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:19:54.976938 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:54.976922 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" (UID: "4c75e242-15d4-4f8f-81ed-b5c07d5b20a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:19:55.075392 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.075357 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4v4\" (UniqueName: \"kubernetes.io/projected/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-kube-api-access-wh4v4\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:19:55.075392 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.075387 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:19:55.075392 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.075397 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:19:55.347180 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.347141 2556 generic.go:356] "Generic (PLEG): container finished" podID="4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" containerID="587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782" exitCode=0 Jul 27 22:19:55.347356 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.347187 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" Jul 27 22:19:55.347356 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.347229 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" event={"ID":"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7","Type":"ContainerDied","Data":"587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782"} Jul 27 22:19:55.347356 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.347272 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs" event={"ID":"4c75e242-15d4-4f8f-81ed-b5c07d5b20a7","Type":"ContainerDied","Data":"01f4f1d4748f71316b5eaf182cbed1bfaf3107983142399468af4e3a279820da"} Jul 27 22:19:55.347356 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.347295 2556 scope.go:117] "RemoveContainer" containerID="587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782" Jul 27 22:19:55.357139 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.357117 2556 scope.go:117] "RemoveContainer" containerID="587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782" Jul 27 22:19:55.357428 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:19:55.357409 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782\": container with ID starting with 587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782 not found: ID does not exist" containerID="587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782" Jul 27 22:19:55.357491 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.357441 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782"} err="failed to get container status \"587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782\": rpc error: code = NotFound desc = could not find container \"587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782\": container with ID starting with 587ca2b42a6db5af608111bb4a5e17ccce4531049cebcc07d6169fd6476ad782 not found: ID does not exist" Jul 27 22:19:55.368517 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.368485 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs"] Jul 27 22:19:55.372169 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:55.372144 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/auth-test-model-94450e-kserve-5b48589bd7-k6dxs"] Jul 27 22:19:56.192740 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:56.192696 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" path="/var/lib/kubelet/pods/4c75e242-15d4-4f8f-81ed-b5c07d5b20a7/volumes" Jul 27 22:19:58.604784 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:58.604746 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:58.605202 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:58.604912 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:58.617836 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:58.617808 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:19:59.385615 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:19:59.385586 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:20:06.496973 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:20:06.496936 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:20:09.994558 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:20:09.994523 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:21:03.092361 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:03.092324 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:21:16.405280 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:16.405238 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:21:28.913067 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:28.913032 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn"] Jul 27 22:21:28.970571 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:28.970533 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:29.731349 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:29.731315 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:31.891572 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:31.891537 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:21:43.932248 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:43.932086 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" podUID="69274bfc-81d9-44a0-aaf4-1301c8070f75" containerName="main" containerID="cri-o://ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695" gracePeriod=45 Jul 27 22:21:44.186132 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.186064 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:21:44.337108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337072 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-model-cache\") pod \"69274bfc-81d9-44a0-aaf4-1301c8070f75\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " Jul 27 22:21:44.337293 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337119 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f4cv\" (UniqueName: \"kubernetes.io/projected/69274bfc-81d9-44a0-aaf4-1301c8070f75-kube-api-access-9f4cv\") pod \"69274bfc-81d9-44a0-aaf4-1301c8070f75\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " Jul 27 22:21:44.337293 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337140 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-tmp-dir\") pod \"69274bfc-81d9-44a0-aaf4-1301c8070f75\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " Jul 27 22:21:44.337293 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337160 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-dshm\") pod \"69274bfc-81d9-44a0-aaf4-1301c8070f75\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " Jul 27 22:21:44.337293 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337209 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69274bfc-81d9-44a0-aaf4-1301c8070f75-tls-certs\") pod \"69274bfc-81d9-44a0-aaf4-1301c8070f75\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " Jul 27 22:21:44.337293 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337279 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-home\") pod \"69274bfc-81d9-44a0-aaf4-1301c8070f75\" (UID: \"69274bfc-81d9-44a0-aaf4-1301c8070f75\") " Jul 27 22:21:44.337561 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337424 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-model-cache" (OuterVolumeSpecName: "model-cache") pod "69274bfc-81d9-44a0-aaf4-1301c8070f75" (UID: "69274bfc-81d9-44a0-aaf4-1301c8070f75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:21:44.337561 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337434 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69274bfc-81d9-44a0-aaf4-1301c8070f75" (UID: "69274bfc-81d9-44a0-aaf4-1301c8070f75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:21:44.337696 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337582 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:21:44.337696 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337601 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:21:44.337696 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.337613 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-home" (OuterVolumeSpecName: "home") pod "69274bfc-81d9-44a0-aaf4-1301c8070f75" (UID: "69274bfc-81d9-44a0-aaf4-1301c8070f75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:21:44.339513 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.339486 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-dshm" (OuterVolumeSpecName: "dshm") pod "69274bfc-81d9-44a0-aaf4-1301c8070f75" (UID: "69274bfc-81d9-44a0-aaf4-1301c8070f75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:21:44.339623 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.339557 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69274bfc-81d9-44a0-aaf4-1301c8070f75-kube-api-access-9f4cv" (OuterVolumeSpecName: "kube-api-access-9f4cv") pod "69274bfc-81d9-44a0-aaf4-1301c8070f75" (UID: "69274bfc-81d9-44a0-aaf4-1301c8070f75"). InnerVolumeSpecName "kube-api-access-9f4cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:21:44.339623 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.339578 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69274bfc-81d9-44a0-aaf4-1301c8070f75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69274bfc-81d9-44a0-aaf4-1301c8070f75" (UID: "69274bfc-81d9-44a0-aaf4-1301c8070f75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:21:44.438106 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.438019 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:21:44.438106 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.438050 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f4cv\" (UniqueName: \"kubernetes.io/projected/69274bfc-81d9-44a0-aaf4-1301c8070f75-kube-api-access-9f4cv\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:21:44.438106 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.438061 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69274bfc-81d9-44a0-aaf4-1301c8070f75-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:21:44.438106 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.438069 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69274bfc-81d9-44a0-aaf4-1301c8070f75-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:21:44.922981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.922945 2556 generic.go:356] "Generic (PLEG): container finished" podID="69274bfc-81d9-44a0-aaf4-1301c8070f75" containerID="ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695" exitCode=0 Jul 27 22:21:44.923176 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.922994 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" event={"ID":"69274bfc-81d9-44a0-aaf4-1301c8070f75","Type":"ContainerDied","Data":"ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695"} Jul 27 22:21:44.923176 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.923002 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" Jul 27 22:21:44.923176 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.923021 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn" event={"ID":"69274bfc-81d9-44a0-aaf4-1301c8070f75","Type":"ContainerDied","Data":"fa140114532e3c1dcd1bbfd8dadf0e2eb9f284ed3d3c23e8ff9e236a68400404"} Jul 27 22:21:44.923176 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.923040 2556 scope.go:117] "RemoveContainer" containerID="ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695" Jul 27 22:21:44.932552 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.932529 2556 scope.go:117] "RemoveContainer" containerID="ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695" Jul 27 22:21:44.932982 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:21:44.932962 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695\": container with ID starting with ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695 not found: ID does not exist" containerID="ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695" Jul 27 22:21:44.933027 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.932995 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695"} err="failed to get container status \"ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695\": rpc error: code = NotFound desc = could not find container \"ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695\": container with ID starting with ca727141619758e73817ccb2e0f995e8b5e63a512af1e2bba21877063796e695 not found: ID does not exist" Jul 27 22:21:44.947749 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.947647 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn"] Jul 27 22:21:44.955387 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:44.955359 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/sub-test-model-94450e-kserve-7578bc7fb5-h28dn"] Jul 27 22:21:45.392285 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:45.392248 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:21:46.192507 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:46.192468 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69274bfc-81d9-44a0-aaf4-1301c8070f75" path="/var/lib/kubelet/pods/69274bfc-81d9-44a0-aaf4-1301c8070f75/volumes" Jul 27 22:21:47.285687 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.285633 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz"] Jul 27 22:21:47.286076 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.285984 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69274bfc-81d9-44a0-aaf4-1301c8070f75" containerName="main" Jul 27 22:21:47.286076 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.285997 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="69274bfc-81d9-44a0-aaf4-1301c8070f75" containerName="main" Jul 27 22:21:47.286076 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.286012 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" containerName="main" Jul 27 22:21:47.286076 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.286018 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" containerName="main" Jul 27 22:21:47.286076 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.286075 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="69274bfc-81d9-44a0-aaf4-1301c8070f75" containerName="main" Jul 27 22:21:47.286275 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.286085 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c75e242-15d4-4f8f-81ed-b5c07d5b20a7" containerName="main" Jul 27 22:21:47.290276 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.290258 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.292926 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.292905 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"default-dockercfg-f9zxb\"" Jul 27 22:21:47.293042 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.292966 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"kube-root-ca.crt\"" Jul 27 22:21:47.293892 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.293875 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"openshift-service-ca.crt\"" Jul 27 22:21:47.293972 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.293955 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-94450e\"/\"rate-test-model-94450e-kserve-self-signed-certs\"" Jul 27 22:21:47.300371 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.300350 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz"] Jul 27 22:21:47.311818 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.311776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:21:47.365603 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.365563 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-model-cache\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.365805 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.365611 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-tmp-dir\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.365805 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.365751 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-home\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.365805 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.365780 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b044f4-279c-4d54-aa73-8b328c411766-tls-certs\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.365913 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.365857 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpsd9\" (UniqueName: \"kubernetes.io/projected/e7b044f4-279c-4d54-aa73-8b328c411766-kube-api-access-bpsd9\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.365913 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.365875 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-dshm\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.466902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.466860 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-home\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.466902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.466904 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b044f4-279c-4d54-aa73-8b328c411766-tls-certs\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.467165 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.466953 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsd9\" (UniqueName: \"kubernetes.io/projected/e7b044f4-279c-4d54-aa73-8b328c411766-kube-api-access-bpsd9\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.467165 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.466978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-dshm\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.467165 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.467146 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-model-cache\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.467322 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.467193 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-tmp-dir\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.467322 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.467274 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-home\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.467550 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.467531 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-model-cache\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.467611 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.467534 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-tmp-dir\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.469473 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.469448 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-dshm\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.469746 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.469729 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b044f4-279c-4d54-aa73-8b328c411766-tls-certs\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.477806 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.477783 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsd9\" (UniqueName: \"kubernetes.io/projected/e7b044f4-279c-4d54-aa73-8b328c411766-kube-api-access-bpsd9\") pod \"rate-test-model-94450e-kserve-bc56cffc5-lnhtz\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.600565 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.600468 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:47.745792 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.745760 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz"] Jul 27 22:21:47.747717 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:21:47.747660 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7b044f4_279c_4d54_aa73_8b328c411766.slice/crio-20835cad9ee794d19b04329da5b3019bf9eac86336b62575da5f8ecbfb6047f4 WatchSource:0}: Error finding container 20835cad9ee794d19b04329da5b3019bf9eac86336b62575da5f8ecbfb6047f4: Status 404 returned error can't find the container with id 20835cad9ee794d19b04329da5b3019bf9eac86336b62575da5f8ecbfb6047f4 Jul 27 22:21:47.945852 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.945817 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" event={"ID":"e7b044f4-279c-4d54-aa73-8b328c411766","Type":"ContainerStarted","Data":"5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6"} Jul 27 22:21:47.946023 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.945864 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" event={"ID":"e7b044f4-279c-4d54-aa73-8b328c411766","Type":"ContainerStarted","Data":"20835cad9ee794d19b04329da5b3019bf9eac86336b62575da5f8ecbfb6047f4"} Jul 27 22:21:47.967368 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:47.967316 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" podStartSLOduration=0.967301181 podStartE2EDuration="967.301181ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:21:47.964579423 +0000 UTC m=+2876.373001453" watchObservedRunningTime="2026-07-27 22:21:47.967301181 +0000 UTC m=+2876.375723210" Jul 27 22:21:57.497261 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:57.497218 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:21:57.600608 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:57.600574 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:57.600832 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:57.600622 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:57.613407 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:57.613377 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:21:58.007847 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:21:58.007819 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:22:15.334608 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:22:15.334564 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:19.287438 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:22:19.287404 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:22:24.943101 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:22:24.943068 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:25.051032 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:22:25.050977 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:41.787619 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:22:41.787583 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:22:48.629326 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:22:48.629286 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:23:04.101426 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:04.101384 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:23:13.310598 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:13.310476 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:13.411309 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:13.411276 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:22.093680 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:22.093645 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:23:25.847084 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:25.847045 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz"] Jul 27 22:23:25.861511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:25.861467 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:26.685018 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:26.684980 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:27.645470 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.645430 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7805-140198078009136"] Jul 27 22:23:27.648822 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.648805 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:27.651535 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.651509 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8k7th\"" Jul 27 22:23:27.654816 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.654794 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198078009136"] Jul 27 22:23:27.746313 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.746279 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l22b4\" (UniqueName: \"kubernetes.io/projected/8d85ff10-6f3e-4226-84f4-359d9a343f5b-kube-api-access-l22b4\") pod \"test-curl-7805-140198078009136\" (UID: \"8d85ff10-6f3e-4226-84f4-359d9a343f5b\") " pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:27.847118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.847085 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l22b4\" (UniqueName: \"kubernetes.io/projected/8d85ff10-6f3e-4226-84f4-359d9a343f5b-kube-api-access-l22b4\") pod \"test-curl-7805-140198078009136\" (UID: \"8d85ff10-6f3e-4226-84f4-359d9a343f5b\") " pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:27.856158 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.856126 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l22b4\" (UniqueName: \"kubernetes.io/projected/8d85ff10-6f3e-4226-84f4-359d9a343f5b-kube-api-access-l22b4\") pod \"test-curl-7805-140198078009136\" (UID: \"8d85ff10-6f3e-4226-84f4-359d9a343f5b\") " pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:27.959345 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:27.959312 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:28.096604 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:28.096583 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198078009136"] Jul 27 22:23:28.098891 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:23:28.098863 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d85ff10_6f3e_4226_84f4_359d9a343f5b.slice/crio-780bede839a768bcfa2e02b5a5ff08d2c88a2d4e3c387f5b6860ebe51b513814 WatchSource:0}: Error finding container 780bede839a768bcfa2e02b5a5ff08d2c88a2d4e3c387f5b6860ebe51b513814: Status 404 returned error can't find the container with id 780bede839a768bcfa2e02b5a5ff08d2c88a2d4e3c387f5b6860ebe51b513814 Jul 27 22:23:28.100300 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:28.100282 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 22:23:28.463368 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:28.463330 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198078009136" event={"ID":"8d85ff10-6f3e-4226-84f4-359d9a343f5b","Type":"ContainerStarted","Data":"780bede839a768bcfa2e02b5a5ff08d2c88a2d4e3c387f5b6860ebe51b513814"} Jul 27 22:23:29.472551 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:29.472514 2556 generic.go:356] "Generic (PLEG): container finished" podID="8d85ff10-6f3e-4226-84f4-359d9a343f5b" containerID="195d8d71e0c630daa701f96414cc14ee54cec2366eeae80cf6640d3ffb01e6f9" exitCode=0 Jul 27 22:23:29.473025 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:29.472602 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198078009136" event={"ID":"8d85ff10-6f3e-4226-84f4-359d9a343f5b","Type":"ContainerDied","Data":"195d8d71e0c630daa701f96414cc14ee54cec2366eeae80cf6640d3ffb01e6f9"} Jul 27 22:23:30.612837 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.612809 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:30.636075 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.636041 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7805-140198078009136_8d85ff10-6f3e-4226-84f4-359d9a343f5b/test-curl-7805-140198078009136/0.log" Jul 27 22:23:30.670874 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.670846 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l22b4\" (UniqueName: \"kubernetes.io/projected/8d85ff10-6f3e-4226-84f4-359d9a343f5b-kube-api-access-l22b4\") pod \"8d85ff10-6f3e-4226-84f4-359d9a343f5b\" (UID: \"8d85ff10-6f3e-4226-84f4-359d9a343f5b\") " Jul 27 22:23:30.673230 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.673201 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d85ff10-6f3e-4226-84f4-359d9a343f5b-kube-api-access-l22b4" (OuterVolumeSpecName: "kube-api-access-l22b4") pod "8d85ff10-6f3e-4226-84f4-359d9a343f5b" (UID: "8d85ff10-6f3e-4226-84f4-359d9a343f5b"). InnerVolumeSpecName "kube-api-access-l22b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:30.676987 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.676965 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7805-140198078009136"] Jul 27 22:23:30.685735 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.685709 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7805-140198078009136"] Jul 27 22:23:30.771867 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.771779 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l22b4\" (UniqueName: \"kubernetes.io/projected/8d85ff10-6f3e-4226-84f4-359d9a343f5b-kube-api-access-l22b4\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:30.831471 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.831433 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7805-140198078005808"] Jul 27 22:23:30.831855 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.831838 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d85ff10-6f3e-4226-84f4-359d9a343f5b" containerName="test-curl-7805-140198078009136" Jul 27 22:23:30.831902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.831857 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d85ff10-6f3e-4226-84f4-359d9a343f5b" containerName="test-curl-7805-140198078009136" Jul 27 22:23:30.831939 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.831925 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d85ff10-6f3e-4226-84f4-359d9a343f5b" containerName="test-curl-7805-140198078009136" Jul 27 22:23:30.834880 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.834863 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078005808" Jul 27 22:23:30.850856 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.850826 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198078005808"] Jul 27 22:23:30.973127 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:30.973092 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fck4\" (UniqueName: \"kubernetes.io/projected/4eca576b-92cf-4096-8de1-55e93485b230-kube-api-access-5fck4\") pod \"test-curl-7805-140198078005808\" (UID: \"4eca576b-92cf-4096-8de1-55e93485b230\") " pod="opendatahub/test-curl-7805-140198078005808" Jul 27 22:23:31.074009 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:31.073875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fck4\" (UniqueName: \"kubernetes.io/projected/4eca576b-92cf-4096-8de1-55e93485b230-kube-api-access-5fck4\") pod \"test-curl-7805-140198078005808\" (UID: \"4eca576b-92cf-4096-8de1-55e93485b230\") " pod="opendatahub/test-curl-7805-140198078005808" Jul 27 22:23:31.083047 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:31.083020 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fck4\" (UniqueName: \"kubernetes.io/projected/4eca576b-92cf-4096-8de1-55e93485b230-kube-api-access-5fck4\") pod \"test-curl-7805-140198078005808\" (UID: \"4eca576b-92cf-4096-8de1-55e93485b230\") " pod="opendatahub/test-curl-7805-140198078005808" Jul 27 22:23:31.143991 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:31.143963 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078005808" Jul 27 22:23:31.280474 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:31.280449 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198078005808"] Jul 27 22:23:31.282201 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:23:31.282171 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eca576b_92cf_4096_8de1_55e93485b230.slice/crio-458c3516d6074b2bb010b4e2fea0ce3b307009a5cc23bba9fe2ae93205066bd9 WatchSource:0}: Error finding container 458c3516d6074b2bb010b4e2fea0ce3b307009a5cc23bba9fe2ae93205066bd9: Status 404 returned error can't find the container with id 458c3516d6074b2bb010b4e2fea0ce3b307009a5cc23bba9fe2ae93205066bd9 Jul 27 22:23:31.483726 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:31.483693 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:31.483726 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:31.483718 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="780bede839a768bcfa2e02b5a5ff08d2c88a2d4e3c387f5b6860ebe51b513814" Jul 27 22:23:31.484981 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:31.484954 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198078005808" event={"ID":"4eca576b-92cf-4096-8de1-55e93485b230","Type":"ContainerStarted","Data":"458c3516d6074b2bb010b4e2fea0ce3b307009a5cc23bba9fe2ae93205066bd9"} Jul 27 22:23:31.494434 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:23:31.494409 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7805-140198078009136\" is forbidden: User \"system:node:ip-10-0-133-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-37.ec2.internal' and this object" podUID="8d85ff10-6f3e-4226-84f4-359d9a343f5b" pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:32.191875 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:23:32.191837 2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7805-140198078009136\" is forbidden: User \"system:node:ip-10-0-133-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-37.ec2.internal' and this object" podUID="8d85ff10-6f3e-4226-84f4-359d9a343f5b" pod="opendatahub/test-curl-7805-140198078009136" Jul 27 22:23:32.192433 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:32.192410 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d85ff10-6f3e-4226-84f4-359d9a343f5b" path="/var/lib/kubelet/pods/8d85ff10-6f3e-4226-84f4-359d9a343f5b/volumes" Jul 27 22:23:32.491995 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:32.491960 2556 generic.go:356] "Generic (PLEG): container finished" podID="4eca576b-92cf-4096-8de1-55e93485b230" containerID="0a4362dc3ac9dbe5fc8f0ced2e130a8fdf8c3cdc6c724e090d957e3aaefbe755" exitCode=0 Jul 27 22:23:32.492173 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:32.492047 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198078005808" event={"ID":"4eca576b-92cf-4096-8de1-55e93485b230","Type":"ContainerDied","Data":"0a4362dc3ac9dbe5fc8f0ced2e130a8fdf8c3cdc6c724e090d957e3aaefbe755"} Jul 27 22:23:33.289843 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.289805 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:33.389815 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.389776 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:33.631618 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.631593 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078005808" Jul 27 22:23:33.653772 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.653744 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7805-140198078005808_4eca576b-92cf-4096-8de1-55e93485b230/test-curl-7805-140198078005808/0.log" Jul 27 22:23:33.679486 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.679450 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7805-140198078005808"] Jul 27 22:23:33.685847 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.685823 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7805-140198078005808"] Jul 27 22:23:33.698886 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.698864 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fck4\" (UniqueName: \"kubernetes.io/projected/4eca576b-92cf-4096-8de1-55e93485b230-kube-api-access-5fck4\") pod \"4eca576b-92cf-4096-8de1-55e93485b230\" (UID: \"4eca576b-92cf-4096-8de1-55e93485b230\") " Jul 27 22:23:33.701173 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.701141 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eca576b-92cf-4096-8de1-55e93485b230-kube-api-access-5fck4" (OuterVolumeSpecName: "kube-api-access-5fck4") pod "4eca576b-92cf-4096-8de1-55e93485b230" (UID: "4eca576b-92cf-4096-8de1-55e93485b230"). InnerVolumeSpecName "kube-api-access-5fck4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:33.799536 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.799502 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fck4\" (UniqueName: \"kubernetes.io/projected/4eca576b-92cf-4096-8de1-55e93485b230-kube-api-access-5fck4\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:33.853562 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.853485 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:33.853877 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.853863 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eca576b-92cf-4096-8de1-55e93485b230" containerName="test-curl-7805-140198078005808" Jul 27 22:23:33.853930 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.853879 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eca576b-92cf-4096-8de1-55e93485b230" containerName="test-curl-7805-140198078005808" Jul 27 22:23:33.853970 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.853935 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eca576b-92cf-4096-8de1-55e93485b230" containerName="test-curl-7805-140198078005808" Jul 27 22:23:33.856898 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.856879 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:33.866321 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:33.866292 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:34.001308 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.001272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6d5\" (UniqueName: \"kubernetes.io/projected/34491a1a-922c-4b92-9fe2-7bfa1e97bbd5-kube-api-access-bx6d5\") pod \"test-curl-7805-140198075746032\" (UID: \"34491a1a-922c-4b92-9fe2-7bfa1e97bbd5\") " pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:34.102894 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.102856 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6d5\" (UniqueName: \"kubernetes.io/projected/34491a1a-922c-4b92-9fe2-7bfa1e97bbd5-kube-api-access-bx6d5\") pod \"test-curl-7805-140198075746032\" (UID: \"34491a1a-922c-4b92-9fe2-7bfa1e97bbd5\") " pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:34.112141 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.112066 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6d5\" (UniqueName: \"kubernetes.io/projected/34491a1a-922c-4b92-9fe2-7bfa1e97bbd5-kube-api-access-bx6d5\") pod \"test-curl-7805-140198075746032\" (UID: \"34491a1a-922c-4b92-9fe2-7bfa1e97bbd5\") " pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:34.166554 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.166518 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:34.193251 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.193221 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eca576b-92cf-4096-8de1-55e93485b230" path="/var/lib/kubelet/pods/4eca576b-92cf-4096-8de1-55e93485b230/volumes" Jul 27 22:23:34.315394 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.311795 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:34.503912 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.503874 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075746032" event={"ID":"34491a1a-922c-4b92-9fe2-7bfa1e97bbd5","Type":"ContainerStarted","Data":"9d53b65281ec13d0373406a36d28d1fc416062ea9d3406ce507cd1b306efea82"} Jul 27 22:23:34.505141 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.505120 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198078005808" Jul 27 22:23:34.505278 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:34.505127 2556 scope.go:117] "RemoveContainer" containerID="0a4362dc3ac9dbe5fc8f0ced2e130a8fdf8c3cdc6c724e090d957e3aaefbe755" Jul 27 22:23:35.511593 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:35.511560 2556 generic.go:356] "Generic (PLEG): container finished" podID="34491a1a-922c-4b92-9fe2-7bfa1e97bbd5" containerID="62cf0f93fed35cb3cda461cb312fd242272a4ec188c768e4a3347beb7abea320" exitCode=0 Jul 27 22:23:35.512037 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:35.511721 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075746032" event={"ID":"34491a1a-922c-4b92-9fe2-7bfa1e97bbd5","Type":"ContainerDied","Data":"62cf0f93fed35cb3cda461cb312fd242272a4ec188c768e4a3347beb7abea320"} Jul 27 22:23:36.653049 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.653025 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:36.676773 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.676717 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7805-140198075746032_34491a1a-922c-4b92-9fe2-7bfa1e97bbd5/test-curl-7805-140198075746032/0.log" Jul 27 22:23:36.706007 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.705973 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:36.713606 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.713574 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:36.724758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.724734 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx6d5\" (UniqueName: \"kubernetes.io/projected/34491a1a-922c-4b92-9fe2-7bfa1e97bbd5-kube-api-access-bx6d5\") pod \"34491a1a-922c-4b92-9fe2-7bfa1e97bbd5\" (UID: \"34491a1a-922c-4b92-9fe2-7bfa1e97bbd5\") " Jul 27 22:23:36.727017 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.726990 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34491a1a-922c-4b92-9fe2-7bfa1e97bbd5-kube-api-access-bx6d5" (OuterVolumeSpecName: "kube-api-access-bx6d5") pod "34491a1a-922c-4b92-9fe2-7bfa1e97bbd5" (UID: "34491a1a-922c-4b92-9fe2-7bfa1e97bbd5"). InnerVolumeSpecName "kube-api-access-bx6d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:36.826043 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.825944 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx6d5\" (UniqueName: \"kubernetes.io/projected/34491a1a-922c-4b92-9fe2-7bfa1e97bbd5-kube-api-access-bx6d5\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:36.845624 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.845593 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:36.845972 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.845960 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34491a1a-922c-4b92-9fe2-7bfa1e97bbd5" containerName="test-curl-7805-140198075746032" Jul 27 22:23:36.846020 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.845974 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="34491a1a-922c-4b92-9fe2-7bfa1e97bbd5" containerName="test-curl-7805-140198075746032" Jul 27 22:23:36.846065 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.846055 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="34491a1a-922c-4b92-9fe2-7bfa1e97bbd5" containerName="test-curl-7805-140198075746032" Jul 27 22:23:36.848640 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.848623 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:36.856639 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.856614 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:36.927072 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:36.927039 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2vnk\" (UniqueName: \"kubernetes.io/projected/4ad9673e-952e-456f-a3b1-3283d2c94ddc-kube-api-access-t2vnk\") pod \"test-curl-7805-140198075746032\" (UID: \"4ad9673e-952e-456f-a3b1-3283d2c94ddc\") " pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:37.027529 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.027489 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2vnk\" (UniqueName: \"kubernetes.io/projected/4ad9673e-952e-456f-a3b1-3283d2c94ddc-kube-api-access-t2vnk\") pod \"test-curl-7805-140198075746032\" (UID: \"4ad9673e-952e-456f-a3b1-3283d2c94ddc\") " pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:37.039417 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.039382 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2vnk\" (UniqueName: \"kubernetes.io/projected/4ad9673e-952e-456f-a3b1-3283d2c94ddc-kube-api-access-t2vnk\") pod \"test-curl-7805-140198075746032\" (UID: \"4ad9673e-952e-456f-a3b1-3283d2c94ddc\") " pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:37.158872 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.158781 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:37.300056 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.299980 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:37.302145 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:23:37.302122 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ad9673e_952e_456f_a3b1_3283d2c94ddc.slice/crio-4f4f35fa860af67eb593627a0aea17935f43a00bfa9fab6173d536efbcd3ed35 WatchSource:0}: Error finding container 4f4f35fa860af67eb593627a0aea17935f43a00bfa9fab6173d536efbcd3ed35: Status 404 returned error can't find the container with id 4f4f35fa860af67eb593627a0aea17935f43a00bfa9fab6173d536efbcd3ed35 Jul 27 22:23:37.524223 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.524185 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075746032" event={"ID":"4ad9673e-952e-456f-a3b1-3283d2c94ddc","Type":"ContainerStarted","Data":"4f4f35fa860af67eb593627a0aea17935f43a00bfa9fab6173d536efbcd3ed35"} Jul 27 22:23:37.525394 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.525372 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:37.525394 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.525386 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d53b65281ec13d0373406a36d28d1fc416062ea9d3406ce507cd1b306efea82" Jul 27 22:23:37.537159 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:37.537125 2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7805-140198075746032" oldPodUID="34491a1a-922c-4b92-9fe2-7bfa1e97bbd5" podUID="4ad9673e-952e-456f-a3b1-3283d2c94ddc" Jul 27 22:23:38.106846 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:38.106811 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:23:38.192406 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:38.192371 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34491a1a-922c-4b92-9fe2-7bfa1e97bbd5" path="/var/lib/kubelet/pods/34491a1a-922c-4b92-9fe2-7bfa1e97bbd5/volumes" Jul 27 22:23:38.532725 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:38.532661 2556 generic.go:356] "Generic (PLEG): container finished" podID="4ad9673e-952e-456f-a3b1-3283d2c94ddc" containerID="d9677c2c081a37765cf531ff9b1300b7abc7b5b87cc56b914ad8e0a21c53d060" exitCode=0 Jul 27 22:23:38.532725 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:38.532712 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075746032" event={"ID":"4ad9673e-952e-456f-a3b1-3283d2c94ddc","Type":"ContainerDied","Data":"d9677c2c081a37765cf531ff9b1300b7abc7b5b87cc56b914ad8e0a21c53d060"} Jul 27 22:23:39.676271 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.676244 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:39.696757 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.696728 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7805-140198075746032_4ad9673e-952e-456f-a3b1-3283d2c94ddc/test-curl-7805-140198075746032/0.log" Jul 27 22:23:39.726453 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.725317 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:39.730318 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.730280 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7805-140198075746032"] Jul 27 22:23:39.753465 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.753434 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2vnk\" (UniqueName: \"kubernetes.io/projected/4ad9673e-952e-456f-a3b1-3283d2c94ddc-kube-api-access-t2vnk\") pod \"4ad9673e-952e-456f-a3b1-3283d2c94ddc\" (UID: \"4ad9673e-952e-456f-a3b1-3283d2c94ddc\") " Jul 27 22:23:39.755706 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.755665 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad9673e-952e-456f-a3b1-3283d2c94ddc-kube-api-access-t2vnk" (OuterVolumeSpecName: "kube-api-access-t2vnk") pod "4ad9673e-952e-456f-a3b1-3283d2c94ddc" (UID: "4ad9673e-952e-456f-a3b1-3283d2c94ddc"). InnerVolumeSpecName "kube-api-access-t2vnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:39.854695 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.854590 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2vnk\" (UniqueName: \"kubernetes.io/projected/4ad9673e-952e-456f-a3b1-3283d2c94ddc-kube-api-access-t2vnk\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:39.857297 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.857270 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:39.857688 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.857659 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad9673e-952e-456f-a3b1-3283d2c94ddc" containerName="test-curl-7805-140198075746032" Jul 27 22:23:39.857739 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.857691 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad9673e-952e-456f-a3b1-3283d2c94ddc" containerName="test-curl-7805-140198075746032" Jul 27 22:23:39.857783 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.857774 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad9673e-952e-456f-a3b1-3283d2c94ddc" containerName="test-curl-7805-140198075746032" Jul 27 22:23:39.860766 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.860750 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:39.867342 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.867311 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:39.955773 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:39.955732 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68f4\" (UniqueName: \"kubernetes.io/projected/369022c8-ab0f-45c9-b4a7-412cc9c42beb-kube-api-access-c68f4\") pod \"test-curl-7805-140198075747040\" (UID: \"369022c8-ab0f-45c9-b4a7-412cc9c42beb\") " pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:40.056545 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.056509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68f4\" (UniqueName: \"kubernetes.io/projected/369022c8-ab0f-45c9-b4a7-412cc9c42beb-kube-api-access-c68f4\") pod \"test-curl-7805-140198075747040\" (UID: \"369022c8-ab0f-45c9-b4a7-412cc9c42beb\") " pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:40.067428 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.067401 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68f4\" (UniqueName: \"kubernetes.io/projected/369022c8-ab0f-45c9-b4a7-412cc9c42beb-kube-api-access-c68f4\") pod \"test-curl-7805-140198075747040\" (UID: \"369022c8-ab0f-45c9-b4a7-412cc9c42beb\") " pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:40.170571 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.170488 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:40.193444 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.193413 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad9673e-952e-456f-a3b1-3283d2c94ddc" path="/var/lib/kubelet/pods/4ad9673e-952e-456f-a3b1-3283d2c94ddc/volumes" Jul 27 22:23:40.307306 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.307280 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:40.308972 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:23:40.308945 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod369022c8_ab0f_45c9_b4a7_412cc9c42beb.slice/crio-5c813f90eb0e95b2e53fa3abb2b33395c6945dd79a670fb795a70bf994e043a2 WatchSource:0}: Error finding container 5c813f90eb0e95b2e53fa3abb2b33395c6945dd79a670fb795a70bf994e043a2: Status 404 returned error can't find the container with id 5c813f90eb0e95b2e53fa3abb2b33395c6945dd79a670fb795a70bf994e043a2 Jul 27 22:23:40.544377 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.544343 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075746032" Jul 27 22:23:40.544569 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.544351 2556 scope.go:117] "RemoveContainer" containerID="d9677c2c081a37765cf531ff9b1300b7abc7b5b87cc56b914ad8e0a21c53d060" Jul 27 22:23:40.545807 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.545588 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075747040" event={"ID":"369022c8-ab0f-45c9-b4a7-412cc9c42beb","Type":"ContainerStarted","Data":"5c813f90eb0e95b2e53fa3abb2b33395c6945dd79a670fb795a70bf994e043a2"} Jul 27 22:23:40.866891 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.866774 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" podUID="e7b044f4-279c-4d54-aa73-8b328c411766" containerName="main" containerID="cri-o://5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6" gracePeriod=45 Jul 27 22:23:40.971616 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:40.971576 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:41.073345 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.073310 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:41.123165 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.123098 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:23:41.268136 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268097 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-model-cache\") pod \"e7b044f4-279c-4d54-aa73-8b328c411766\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " Jul 27 22:23:41.268291 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268158 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b044f4-279c-4d54-aa73-8b328c411766-tls-certs\") pod \"e7b044f4-279c-4d54-aa73-8b328c411766\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " Jul 27 22:23:41.268291 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268180 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-dshm\") pod \"e7b044f4-279c-4d54-aa73-8b328c411766\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " Jul 27 22:23:41.268291 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268257 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-tmp-dir\") pod \"e7b044f4-279c-4d54-aa73-8b328c411766\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " Jul 27 22:23:41.268439 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268301 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-home\") pod \"e7b044f4-279c-4d54-aa73-8b328c411766\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " Jul 27 22:23:41.268439 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268336 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpsd9\" (UniqueName: \"kubernetes.io/projected/e7b044f4-279c-4d54-aa73-8b328c411766-kube-api-access-bpsd9\") pod \"e7b044f4-279c-4d54-aa73-8b328c411766\" (UID: \"e7b044f4-279c-4d54-aa73-8b328c411766\") " Jul 27 22:23:41.268551 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268452 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7b044f4-279c-4d54-aa73-8b328c411766" (UID: "e7b044f4-279c-4d54-aa73-8b328c411766"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.268609 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268585 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7b044f4-279c-4d54-aa73-8b328c411766" (UID: "e7b044f4-279c-4d54-aa73-8b328c411766"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.268660 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268607 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-home" (OuterVolumeSpecName: "home") pod "e7b044f4-279c-4d54-aa73-8b328c411766" (UID: "e7b044f4-279c-4d54-aa73-8b328c411766"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.268753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268714 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.268753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268736 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.268753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.268751 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.270771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.270742 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b044f4-279c-4d54-aa73-8b328c411766-kube-api-access-bpsd9" (OuterVolumeSpecName: "kube-api-access-bpsd9") pod "e7b044f4-279c-4d54-aa73-8b328c411766" (UID: "e7b044f4-279c-4d54-aa73-8b328c411766"). InnerVolumeSpecName "kube-api-access-bpsd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:41.271076 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.271061 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b044f4-279c-4d54-aa73-8b328c411766-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7b044f4-279c-4d54-aa73-8b328c411766" (UID: "e7b044f4-279c-4d54-aa73-8b328c411766"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:23:41.271076 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.271067 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-dshm" (OuterVolumeSpecName: "dshm") pod "e7b044f4-279c-4d54-aa73-8b328c411766" (UID: "e7b044f4-279c-4d54-aa73-8b328c411766"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:23:41.369753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.369721 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpsd9\" (UniqueName: \"kubernetes.io/projected/e7b044f4-279c-4d54-aa73-8b328c411766-kube-api-access-bpsd9\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.369753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.369747 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b044f4-279c-4d54-aa73-8b328c411766-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.369753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.369759 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b044f4-279c-4d54-aa73-8b328c411766-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:41.553103 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.553067 2556 generic.go:356] "Generic (PLEG): container finished" podID="e7b044f4-279c-4d54-aa73-8b328c411766" containerID="5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6" exitCode=0 Jul 27 22:23:41.553305 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.553128 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" Jul 27 22:23:41.553305 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.553147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" event={"ID":"e7b044f4-279c-4d54-aa73-8b328c411766","Type":"ContainerDied","Data":"5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6"} Jul 27 22:23:41.553305 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.553203 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz" event={"ID":"e7b044f4-279c-4d54-aa73-8b328c411766","Type":"ContainerDied","Data":"20835cad9ee794d19b04329da5b3019bf9eac86336b62575da5f8ecbfb6047f4"} Jul 27 22:23:41.553305 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.553230 2556 scope.go:117] "RemoveContainer" containerID="5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6" Jul 27 22:23:41.554875 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.554846 2556 generic.go:356] "Generic (PLEG): container finished" podID="369022c8-ab0f-45c9-b4a7-412cc9c42beb" containerID="e7f8f8add2147e1d181d02701ed4a3959ef2368f48c1224dc6dbb5ef8d9cc448" exitCode=0 Jul 27 22:23:41.554993 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.554957 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075747040" event={"ID":"369022c8-ab0f-45c9-b4a7-412cc9c42beb","Type":"ContainerDied","Data":"e7f8f8add2147e1d181d02701ed4a3959ef2368f48c1224dc6dbb5ef8d9cc448"} Jul 27 22:23:41.563157 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.563125 2556 scope.go:117] "RemoveContainer" containerID="5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6" Jul 27 22:23:41.563473 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:23:41.563453 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6\": container with ID starting with 5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6 not found: ID does not exist" containerID="5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6" Jul 27 22:23:41.563534 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.563484 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6"} err="failed to get container status \"5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6\": rpc error: code = NotFound desc = could not find container \"5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6\": container with ID starting with 5fe024b8474d554232a395ca1b441419a445603b471b358078b189b62dfa62f6 not found: ID does not exist" Jul 27 22:23:41.584801 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.584768 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz"] Jul 27 22:23:41.593493 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:41.593467 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-94450e/rate-test-model-94450e-kserve-bc56cffc5-lnhtz"] Jul 27 22:23:42.193313 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.193277 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b044f4-279c-4d54-aa73-8b328c411766" path="/var/lib/kubelet/pods/e7b044f4-279c-4d54-aa73-8b328c411766/volumes" Jul 27 22:23:42.700821 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.700790 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:42.726847 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.726817 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7805-140198075747040_369022c8-ab0f-45c9-b4a7-412cc9c42beb/test-curl-7805-140198075747040/0.log" Jul 27 22:23:42.753325 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.753295 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:42.755725 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.755702 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:42.783073 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.783041 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c68f4\" (UniqueName: \"kubernetes.io/projected/369022c8-ab0f-45c9-b4a7-412cc9c42beb-kube-api-access-c68f4\") pod \"369022c8-ab0f-45c9-b4a7-412cc9c42beb\" (UID: \"369022c8-ab0f-45c9-b4a7-412cc9c42beb\") " Jul 27 22:23:42.785308 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.785282 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369022c8-ab0f-45c9-b4a7-412cc9c42beb-kube-api-access-c68f4" (OuterVolumeSpecName: "kube-api-access-c68f4") pod "369022c8-ab0f-45c9-b4a7-412cc9c42beb" (UID: "369022c8-ab0f-45c9-b4a7-412cc9c42beb"). InnerVolumeSpecName "kube-api-access-c68f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:42.883736 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.883639 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c68f4\" (UniqueName: \"kubernetes.io/projected/369022c8-ab0f-45c9-b4a7-412cc9c42beb-kube-api-access-c68f4\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.893498 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.894357 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="369022c8-ab0f-45c9-b4a7-412cc9c42beb" containerName="test-curl-7805-140198075747040" Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.894380 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="369022c8-ab0f-45c9-b4a7-412cc9c42beb" containerName="test-curl-7805-140198075747040" Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.894413 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b044f4-279c-4d54-aa73-8b328c411766" containerName="main" Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.894424 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b044f4-279c-4d54-aa73-8b328c411766" containerName="main" Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.894610 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="369022c8-ab0f-45c9-b4a7-412cc9c42beb" containerName="test-curl-7805-140198075747040" Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.894633 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b044f4-279c-4d54-aa73-8b328c411766" containerName="main" Jul 27 22:23:42.899195 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.898463 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:42.904607 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.904575 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:42.984540 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:42.984504 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmrck\" (UniqueName: \"kubernetes.io/projected/5508ca52-f1d7-4285-8422-299c5a36c978-kube-api-access-hmrck\") pod \"test-curl-7805-140198075747040\" (UID: \"5508ca52-f1d7-4285-8422-299c5a36c978\") " pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:43.085409 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.085368 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrck\" (UniqueName: \"kubernetes.io/projected/5508ca52-f1d7-4285-8422-299c5a36c978-kube-api-access-hmrck\") pod \"test-curl-7805-140198075747040\" (UID: \"5508ca52-f1d7-4285-8422-299c5a36c978\") " pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:43.094362 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.094329 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrck\" (UniqueName: \"kubernetes.io/projected/5508ca52-f1d7-4285-8422-299c5a36c978-kube-api-access-hmrck\") pod \"test-curl-7805-140198075747040\" (UID: \"5508ca52-f1d7-4285-8422-299c5a36c978\") " pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:43.210589 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.210556 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:43.347185 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.347160 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:43.349394 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:23:43.349368 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5508ca52_f1d7_4285_8422_299c5a36c978.slice/crio-94efe116fe067e1bc8624ee48009dedfb36ca3687309f80f3aeba651454500fe WatchSource:0}: Error finding container 94efe116fe067e1bc8624ee48009dedfb36ca3687309f80f3aeba651454500fe: Status 404 returned error can't find the container with id 94efe116fe067e1bc8624ee48009dedfb36ca3687309f80f3aeba651454500fe Jul 27 22:23:43.567927 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.567835 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:43.567927 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.567867 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c813f90eb0e95b2e53fa3abb2b33395c6945dd79a670fb795a70bf994e043a2" Jul 27 22:23:43.569098 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.569072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075747040" event={"ID":"5508ca52-f1d7-4285-8422-299c5a36c978","Type":"ContainerStarted","Data":"94efe116fe067e1bc8624ee48009dedfb36ca3687309f80f3aeba651454500fe"} Jul 27 22:23:43.579572 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:43.579546 2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7805-140198075747040" oldPodUID="369022c8-ab0f-45c9-b4a7-412cc9c42beb" podUID="5508ca52-f1d7-4285-8422-299c5a36c978" Jul 27 22:23:44.192299 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:44.192254 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="369022c8-ab0f-45c9-b4a7-412cc9c42beb" path="/var/lib/kubelet/pods/369022c8-ab0f-45c9-b4a7-412cc9c42beb/volumes" Jul 27 22:23:44.575990 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:44.575952 2556 generic.go:356] "Generic (PLEG): container finished" podID="5508ca52-f1d7-4285-8422-299c5a36c978" containerID="a86d38a2f6aa09632947631eb562e9c5a849a798d7abb1e8254fd592c90fe1c3" exitCode=0 Jul 27 22:23:44.576703 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:44.576049 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7805-140198075747040" event={"ID":"5508ca52-f1d7-4285-8422-299c5a36c978","Type":"ContainerDied","Data":"a86d38a2f6aa09632947631eb562e9c5a849a798d7abb1e8254fd592c90fe1c3"} Jul 27 22:23:45.716178 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:45.716153 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:45.747837 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:45.747800 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7805-140198075747040_5508ca52-f1d7-4285-8422-299c5a36c978/test-curl-7805-140198075747040/0.log" Jul 27 22:23:45.779202 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:45.779169 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:45.783025 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:45.782999 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7805-140198075747040"] Jul 27 22:23:45.812137 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:45.812106 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmrck\" (UniqueName: \"kubernetes.io/projected/5508ca52-f1d7-4285-8422-299c5a36c978-kube-api-access-hmrck\") pod \"5508ca52-f1d7-4285-8422-299c5a36c978\" (UID: \"5508ca52-f1d7-4285-8422-299c5a36c978\") " Jul 27 22:23:45.814352 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:45.814322 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5508ca52-f1d7-4285-8422-299c5a36c978-kube-api-access-hmrck" (OuterVolumeSpecName: "kube-api-access-hmrck") pod "5508ca52-f1d7-4285-8422-299c5a36c978" (UID: "5508ca52-f1d7-4285-8422-299c5a36c978"). InnerVolumeSpecName "kube-api-access-hmrck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:23:45.912742 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:45.912648 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmrck\" (UniqueName: \"kubernetes.io/projected/5508ca52-f1d7-4285-8422-299c5a36c978-kube-api-access-hmrck\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:23:46.192787 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.192705 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5508ca52-f1d7-4285-8422-299c5a36c978" path="/var/lib/kubelet/pods/5508ca52-f1d7-4285-8422-299c5a36c978/volumes" Jul 27 22:23:46.304209 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.304171 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:46.589935 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.589904 2556 scope.go:117] "RemoveContainer" containerID="a86d38a2f6aa09632947631eb562e9c5a849a798d7abb1e8254fd592c90fe1c3" Jul 27 22:23:46.590126 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.590089 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7805-140198075747040" Jul 27 22:23:46.634118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.634081 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77"] Jul 27 22:23:46.634489 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.634475 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5508ca52-f1d7-4285-8422-299c5a36c978" containerName="test-curl-7805-140198075747040" Jul 27 22:23:46.634545 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.634492 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5508ca52-f1d7-4285-8422-299c5a36c978" containerName="test-curl-7805-140198075747040" Jul 27 22:23:46.634582 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.634569 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5508ca52-f1d7-4285-8422-299c5a36c978" containerName="test-curl-7805-140198075747040" Jul 27 22:23:46.638958 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.638933 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.639731 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.639705 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:46.644341 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.644322 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dd2178-openshift-default-dockercfg-t57qb\"" Jul 27 22:23:46.652863 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.652834 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77"] Jul 27 22:23:46.821558 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821524 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-data\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.821558 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821564 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-credential-socket\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.822014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821596 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6150c9-f9f0-4157-a504-b616434d341a-istiod-ca-cert\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.822014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821722 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-istio-token\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.822014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821753 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r222s\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-kube-api-access-r222s\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.822014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821847 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-envoy\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.822014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821887 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6150c9-f9f0-4157-a504-b616434d341a-istio-podinfo\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.822014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821917 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-socket\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.822014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.821934 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-certs\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.836138 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.836111 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:46.923275 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923187 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-socket\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923275 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-certs\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923275 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923253 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-data\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923275 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923278 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-credential-socket\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923627 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923310 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6150c9-f9f0-4157-a504-b616434d341a-istiod-ca-cert\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923627 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923337 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-istio-token\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923627 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923352 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r222s\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-kube-api-access-r222s\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923627 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923503 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-envoy\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923627 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923555 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6150c9-f9f0-4157-a504-b616434d341a-istio-podinfo\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923945 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923708 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-certs\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923945 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923800 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-data\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.923945 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923853 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-credential-socket\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.924131 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.923954 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-socket\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.924186 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.924135 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6150c9-f9f0-4157-a504-b616434d341a-istiod-ca-cert\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.926049 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.926031 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-envoy\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.926193 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.926165 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6150c9-f9f0-4157-a504-b616434d341a-istio-podinfo\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.935595 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.935568 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-istio-token\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.938400 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.938376 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r222s\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-kube-api-access-r222s\") pod \"e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:46.949812 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:46.949511 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:47.134342 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.134314 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77"] Jul 27 22:23:47.136546 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:23:47.136509 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6150c9_f9f0_4157_a504_b616434d341a.slice/crio-16a21cc465ba569e52a57004718ee6615114dc73c18c8482838fec0cd005edd2 WatchSource:0}: Error finding container 16a21cc465ba569e52a57004718ee6615114dc73c18c8482838fec0cd005edd2: Status 404 returned error can't find the container with id 16a21cc465ba569e52a57004718ee6615114dc73c18c8482838fec0cd005edd2 Jul 27 22:23:47.138765 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.138736 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:47.138836 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.138801 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:47.138836 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.138828 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:23:47.598207 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.598157 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" event={"ID":"5b6150c9-f9f0-4157-a504-b616434d341a","Type":"ContainerStarted","Data":"749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b"} Jul 27 22:23:47.598406 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.598233 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" event={"ID":"5b6150c9-f9f0-4157-a504-b616434d341a","Type":"ContainerStarted","Data":"16a21cc465ba569e52a57004718ee6615114dc73c18c8482838fec0cd005edd2"} Jul 27 22:23:47.626183 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.626117 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podStartSLOduration=1.626101032 podStartE2EDuration="1.626101032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:23:47.623243728 +0000 UTC m=+2996.031665760" watchObservedRunningTime="2026-07-27 22:23:47.626101032 +0000 UTC m=+2996.034523061" Jul 27 22:23:47.950757 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:47.950721 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:48.955967 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:48.955934 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:49.609961 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:49.609926 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:49.611054 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:49.611030 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:23:49.771663 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:49.771614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:49.864432 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:49.864333 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:50.332465 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:50.332421 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:50.762423 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:50.762389 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:50.857943 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:50.857902 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:50.955836 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:50.955804 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:23:58.199162 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:23:58.199128 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:24:04.356732 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:04.356664 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:05.060647 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.060614 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:05.409009 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.408930 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx"] Jul 27 22:24:05.413334 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.413309 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.421510 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.421483 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-54963f-openshift-default-dockercfg-4t8xn\"" Jul 27 22:24:05.457294 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.457255 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx"] Jul 27 22:24:05.460871 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.460838 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:05.486654 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486620 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-envoy\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.486870 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486663 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fql6\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-kube-api-access-9fql6\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.486870 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486719 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-socket\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.486870 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486746 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47410e32-b2fd-4d9a-a161-78f3410ea164-istiod-ca-cert\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.486870 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486857 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-certs\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.487112 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486877 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-podinfo\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.487112 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-credential-socket\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.487112 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.486946 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-data\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.487112 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.487045 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-token\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.587949 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.587909 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47410e32-b2fd-4d9a-a161-78f3410ea164-istiod-ca-cert\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588162 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.587980 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-certs\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588162 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588007 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-podinfo\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588162 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588034 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-credential-socket\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588162 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-data\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588162 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588157 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-token\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588447 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588232 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-envoy\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588447 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fql6\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-kube-api-access-9fql6\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588447 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588306 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-socket\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588447 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588424 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-certs\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588695 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588499 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-credential-socket\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588766 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588715 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-socket\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.588827 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588767 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-data\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.589009 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.588980 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47410e32-b2fd-4d9a-a161-78f3410ea164-istiod-ca-cert\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.590992 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.590964 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-envoy\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.591383 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.591358 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-podinfo\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.599077 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.599044 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fql6\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-kube-api-access-9fql6\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.599414 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.599391 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-token\") pod \"e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.651922 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.651874 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 22:24:05.652132 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.651978 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 22:24:05.727846 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.727801 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:05.758827 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.758778 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:05.885830 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.885803 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx"] Jul 27 22:24:05.888123 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:24:05.888094 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47410e32_b2fd_4d9a_a161_78f3410ea164.slice/crio-9c62f8f35a44016ae3f9798e444c0d8f166cd643e06ff18f9e88fd92715febc9 WatchSource:0}: Error finding container 9c62f8f35a44016ae3f9798e444c0d8f166cd643e06ff18f9e88fd92715febc9: Status 404 returned error can't find the container with id 9c62f8f35a44016ae3f9798e444c0d8f166cd643e06ff18f9e88fd92715febc9 Jul 27 22:24:05.890427 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.890387 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:05.890534 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.890473 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:05.890534 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:05.890520 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:06.703344 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.703306 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" event={"ID":"47410e32-b2fd-4d9a-a161-78f3410ea164","Type":"ContainerStarted","Data":"b291930fe5972f5f5bdd8dcdbff11629da22e9d7008b108f4ebb0cdee1e82697"} Jul 27 22:24:06.703892 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.703365 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" event={"ID":"47410e32-b2fd-4d9a-a161-78f3410ea164","Type":"ContainerStarted","Data":"9c62f8f35a44016ae3f9798e444c0d8f166cd643e06ff18f9e88fd92715febc9"} Jul 27 22:24:06.704949 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.704925 2556 generic.go:356] "Generic (PLEG): container finished" podID="5b6150c9-f9f0-4157-a504-b616434d341a" containerID="749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b" exitCode=0 Jul 27 22:24:06.705065 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.704995 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" event={"ID":"5b6150c9-f9f0-4157-a504-b616434d341a","Type":"ContainerDied","Data":"749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b"} Jul 27 22:24:06.705318 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.705305 2556 scope.go:117] "RemoveContainer" containerID="749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b" Jul 27 22:24:06.706358 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.706331 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:06.706422 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.706411 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:06.706460 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.706445 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:06.728834 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.728805 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:06.732784 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.732723 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" podStartSLOduration=1.732703418 podStartE2EDuration="1.732703418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:06.729052356 +0000 UTC m=+3015.137474389" watchObservedRunningTime="2026-07-27 22:24:06.732703418 +0000 UTC m=+3015.141125447" Jul 27 22:24:06.734684 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:06.734644 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:07.712778 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:07.712740 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" event={"ID":"5b6150c9-f9f0-4157-a504-b616434d341a","Type":"ContainerStarted","Data":"200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4"} Jul 27 22:24:07.713254 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:07.713054 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:07.713921 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:07.713903 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:07.950736 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:07.950704 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:24:07.950911 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:07.950863 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:07.950956 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:07.950911 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:08.703118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:08.703086 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:08.801518 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:08.801488 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:08.950381 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:08.950347 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:08.950580 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:08.950407 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:09.951184 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:09.951142 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:09.951604 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:09.951203 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:10.661497 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:10.661457 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:10.950693 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:10.950640 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:10.950865 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:10.950735 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:11.136730 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:11.136694 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:11.229225 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:11.229118 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:11.310259 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:11.310216 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:11.950199 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:11.950152 2556 patch_prober.go:28] interesting pod/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 27 22:24:11.950423 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:11.950233 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 27 22:24:12.953796 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:12.953765 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:24:12.954207 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:12.954069 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:24:12.955009 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:12.954989 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:24:13.595492 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:13.595445 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:24:21.810118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:21.810024 2556 generic.go:356] "Generic (PLEG): container finished" podID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerID="b291930fe5972f5f5bdd8dcdbff11629da22e9d7008b108f4ebb0cdee1e82697" exitCode=0 Jul 27 22:24:21.810118 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:21.810101 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" event={"ID":"47410e32-b2fd-4d9a-a161-78f3410ea164","Type":"ContainerDied","Data":"b291930fe5972f5f5bdd8dcdbff11629da22e9d7008b108f4ebb0cdee1e82697"} Jul 27 22:24:21.810685 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:21.810526 2556 scope.go:117] "RemoveContainer" containerID="b291930fe5972f5f5bdd8dcdbff11629da22e9d7008b108f4ebb0cdee1e82697" Jul 27 22:24:21.811659 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:21.811627 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:21.811750 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:21.811734 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:21.811788 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:21.811766 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 27 22:24:22.713220 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:22.713181 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:22.844063 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:22.844021 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" event={"ID":"47410e32-b2fd-4d9a-a161-78f3410ea164","Type":"ContainerStarted","Data":"f10ac6ec2fd0aa752bb6f6a6b44b6acb92707d80bbc5e1ce2626538ce8918fca"} Jul 27 22:24:23.474580 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:23.474537 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:24:23.728254 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:23.728159 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:23.729610 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:23.729578 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:23.729757 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:23.729641 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:24.729115 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:24.729067 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:24.729638 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:24.729141 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:25.728722 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:25.728658 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:25.728954 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:25.728751 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:26.729145 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:26.729082 2556 patch_prober.go:28] interesting pod/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 27 22:24:26.729531 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:26.729175 2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 27 22:24:27.732541 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:27.732509 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:27.733036 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:27.732897 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:24:27.733697 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:24:27.733655 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:25:07.517953 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.514292 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749"] Jul 27 22:25:07.519302 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.519279 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.523055 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.523033 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dd2178\"/\"default-dockercfg-fqmk8\"" Jul 27 22:25:07.524131 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.524108 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dd2178\"/\"openshift-service-ca.crt\"" Jul 27 22:25:07.524331 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.524309 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-dd2178\"/\"kube-root-ca.crt\"" Jul 27 22:25:07.524456 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.524356 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-dd2178\"/\"ipp-route-dd2178-kserve-self-signed-certs\"" Jul 27 22:25:07.540915 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.540884 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749"] Jul 27 22:25:07.575878 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.575847 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-model-cache\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.576062 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.575892 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-dshm\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.576062 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.575912 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tls-certs\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.576062 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.576038 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-home\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.576213 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.576072 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42p26\" (UniqueName: \"kubernetes.io/projected/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-kube-api-access-42p26\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.576213 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.576102 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tmp-dir\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.595357 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.595325 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:07.677152 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677098 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-home\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677152 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677150 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42p26\" (UniqueName: \"kubernetes.io/projected/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-kube-api-access-42p26\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677432 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677332 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tmp-dir\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677432 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677405 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-model-cache\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677549 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677436 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-dshm\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677549 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677450 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tls-certs\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677644 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677575 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-home\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677716 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677645 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tmp-dir\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.677815 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.677794 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-model-cache\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.679876 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.679845 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-dshm\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.680029 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.680014 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tls-certs\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.685489 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.685469 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42p26\" (UniqueName: \"kubernetes.io/projected/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-kube-api-access-42p26\") pod \"ipp-route-dd2178-kserve-579ddf4d8c-84749\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.829972 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.829871 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:07.981928 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:07.981893 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749"] Jul 27 22:25:07.986360 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:25:07.985234 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7df93f4_fe5f_4c0b_bab2_a759f5ece2fb.slice/crio-b44c0f15d26bebf1a95d5ebe344bd3d04a6c380858a9cd2fb8a3e68027ba00c2 WatchSource:0}: Error finding container b44c0f15d26bebf1a95d5ebe344bd3d04a6c380858a9cd2fb8a3e68027ba00c2: Status 404 returned error can't find the container with id b44c0f15d26bebf1a95d5ebe344bd3d04a6c380858a9cd2fb8a3e68027ba00c2 Jul 27 22:25:08.095051 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:08.094965 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" event={"ID":"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb","Type":"ContainerStarted","Data":"26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71"} Jul 27 22:25:08.095051 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:08.095008 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" event={"ID":"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb","Type":"ContainerStarted","Data":"b44c0f15d26bebf1a95d5ebe344bd3d04a6c380858a9cd2fb8a3e68027ba00c2"} Jul 27 22:25:11.094391 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:11.094331 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" podStartSLOduration=4.094311376 podStartE2EDuration="4.094311376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:25:08.125818033 +0000 UTC m=+3076.534240063" watchObservedRunningTime="2026-07-27 22:25:11.094311376 +0000 UTC m=+3079.502733405" Jul 27 22:25:11.096511 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:11.096491 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:25:17.830271 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:17.830232 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:17.830783 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:17.830294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:17.842726 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:17.842698 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:18.165170 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:18.165085 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:25:41.306002 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:41.305967 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:25:54.078752 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:54.078711 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:54.610035 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:54.609998 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:54.769234 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:54.769185 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:57.889566 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:57.889519 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:25:58.010036 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:25:58.010000 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:01.142243 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:01.142196 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:01.599561 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:01.599525 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:01.696256 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:01.696217 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:01.993181 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:01.993149 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:06.901393 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:06.901356 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:26:07.216454 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:07.216413 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:12.601698 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:12.601587 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:30.489279 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:30.489244 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:30.688721 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:30.688645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:30.842990 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:30.842882 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:30.998633 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:30.998572 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:31.327147 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:31.327108 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:35.099172 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:35.099133 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:26:38.137279 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:38.137238 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:38.224169 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:38.224130 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:38.452288 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:38.452260 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:38.738459 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:38.738355 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:53.205045 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:53.205012 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:26:57.279914 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:57.279876 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:57.366262 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:57.366231 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:57.480011 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:57.479969 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:57.581272 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:57.581166 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:26:57.925773 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:26:57.925653 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:02.931307 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:02.931269 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749"] Jul 27 22:27:03.072070 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:03.072033 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:08.610030 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:08.609994 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:27:17.954029 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:17.953983 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" podUID="f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" containerName="main" containerID="cri-o://26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71" gracePeriod=45 Jul 27 22:27:18.208786 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.208719 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:27:18.302441 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302399 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-model-cache\") pod \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " Jul 27 22:27:18.302639 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302471 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tls-certs\") pod \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " Jul 27 22:27:18.302639 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302497 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-dshm\") pod \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " Jul 27 22:27:18.302639 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302576 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-home\") pod \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " Jul 27 22:27:18.302639 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302591 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42p26\" (UniqueName: \"kubernetes.io/projected/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-kube-api-access-42p26\") pod \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " Jul 27 22:27:18.302639 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302605 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tmp-dir\") pod \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\" (UID: \"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb\") " Jul 27 22:27:18.302940 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302727 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" (UID: "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:18.302940 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302886 2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-model-cache\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:27:18.303095 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302939 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-home" (OuterVolumeSpecName: "home") pod "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" (UID: "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:18.303095 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.302960 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" (UID: "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:18.304870 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.304840 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-dshm" (OuterVolumeSpecName: "dshm") pod "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" (UID: "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:27:18.304973 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.304909 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" (UID: "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 22:27:18.304973 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.304912 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-kube-api-access-42p26" (OuterVolumeSpecName: "kube-api-access-42p26") pod "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" (UID: "f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb"). InnerVolumeSpecName "kube-api-access-42p26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:27:18.403733 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.403689 2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-home\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:27:18.403733 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.403728 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42p26\" (UniqueName: \"kubernetes.io/projected/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-kube-api-access-42p26\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:27:18.403733 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.403738 2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tmp-dir\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:27:18.403977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.403747 2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-tls-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:27:18.403977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.403755 2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb-dshm\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:27:18.806929 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.806894 2556 generic.go:356] "Generic (PLEG): container finished" podID="f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" containerID="26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71" exitCode=0 Jul 27 22:27:18.807108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.806945 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" Jul 27 22:27:18.807108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.806987 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" event={"ID":"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb","Type":"ContainerDied","Data":"26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71"} Jul 27 22:27:18.807108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.807037 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" event={"ID":"f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb","Type":"ContainerDied","Data":"b44c0f15d26bebf1a95d5ebe344bd3d04a6c380858a9cd2fb8a3e68027ba00c2"} Jul 27 22:27:18.807108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.807061 2556 scope.go:117] "RemoveContainer" containerID="26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71" Jul 27 22:27:18.817142 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.817122 2556 scope.go:117] "RemoveContainer" containerID="26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71" Jul 27 22:27:18.817415 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:27:18.817396 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71\": container with ID starting with 26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71 not found: ID does not exist" containerID="26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71" Jul 27 22:27:18.817489 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.817426 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71"} err="failed to get container status \"26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71\": rpc error: code = NotFound desc = could not find container \"26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71\": container with ID starting with 26f5db1f653228b8f8140fc1b15238f3da790cdf2cf6e3f67bc97ecc64466f71 not found: ID does not exist" Jul 27 22:27:18.838889 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.838859 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749"] Jul 27 22:27:18.852098 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:18.852067 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749"] Jul 27 22:27:20.192771 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:20.192733 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" path="/var/lib/kubelet/pods/f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb/volumes" Jul 27 22:27:23.155254 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:23.155206 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-dd2178/ipp-route-dd2178-kserve-579ddf4d8c-84749" podUID="f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 27 22:27:40.060302 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:40.060264 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:41.497382 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:41.497346 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:47.327019 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:47.326922 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:27:50.088098 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:27:50.088064 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:04.368788 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:04.368750 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:04.461267 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:04.461233 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:04.632839 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:04.632757 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77"] Jul 27 22:28:04.633078 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:04.633027 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" containerID="cri-o://200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4" gracePeriod=30 Jul 27 22:28:04.657753 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:04.657721 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:04.924282 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:04.924174 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:09.878361 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.878329 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:28:09.987808 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.987776 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-envoy\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988026 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.987816 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-credential-socket\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988026 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.987853 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-socket\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988026 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.987877 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6150c9-f9f0-4157-a504-b616434d341a-istiod-ca-cert\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988026 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.987914 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6150c9-f9f0-4157-a504-b616434d341a-istio-podinfo\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988026 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.987944 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-istio-token\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988026 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.987963 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-certs\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988357 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988111 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r222s\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-kube-api-access-r222s\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988357 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988159 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-data\") pod \"5b6150c9-f9f0-4157-a504-b616434d341a\" (UID: \"5b6150c9-f9f0-4157-a504-b616434d341a\") " Jul 27 22:28:09.988357 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988142 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:09.988357 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988216 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:09.988357 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988313 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:09.988618 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988474 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:09.988618 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988495 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:09.988618 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988537 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:09.988618 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.988469 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6150c9-f9f0-4157-a504-b616434d341a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:28:09.989391 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.989362 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:09.990312 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.990282 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:09.990312 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.990299 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:28:09.990466 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.990449 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-kube-api-access-r222s" (OuterVolumeSpecName: "kube-api-access-r222s") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "kube-api-access-r222s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:28:09.990507 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:09.990482 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b6150c9-f9f0-4157-a504-b616434d341a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b6150c9-f9f0-4157-a504-b616434d341a" (UID: "5b6150c9-f9f0-4157-a504-b616434d341a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:28:10.089623 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.089590 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r222s\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-kube-api-access-r222s\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:10.089623 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.089620 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:10.089623 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.089630 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b6150c9-f9f0-4157-a504-b616434d341a-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:10.089921 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.089638 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b6150c9-f9f0-4157-a504-b616434d341a-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:10.089921 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.089646 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b6150c9-f9f0-4157-a504-b616434d341a-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:10.089921 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.089654 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b6150c9-f9f0-4157-a504-b616434d341a-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:10.093229 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.093200 2556 generic.go:356] "Generic (PLEG): container finished" podID="5b6150c9-f9f0-4157-a504-b616434d341a" containerID="200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4" exitCode=0 Jul 27 22:28:10.093380 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.093259 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" Jul 27 22:28:10.093380 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.093287 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" event={"ID":"5b6150c9-f9f0-4157-a504-b616434d341a","Type":"ContainerDied","Data":"200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4"} Jul 27 22:28:10.093380 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.093344 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77" event={"ID":"5b6150c9-f9f0-4157-a504-b616434d341a","Type":"ContainerDied","Data":"16a21cc465ba569e52a57004718ee6615114dc73c18c8482838fec0cd005edd2"} Jul 27 22:28:10.093380 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.093366 2556 scope.go:117] "RemoveContainer" containerID="200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4" Jul 27 22:28:10.103540 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.103521 2556 scope.go:117] "RemoveContainer" containerID="749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b" Jul 27 22:28:10.113020 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.112998 2556 scope.go:117] "RemoveContainer" containerID="200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4" Jul 27 22:28:10.113328 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:28:10.113306 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4\": container with ID starting with 200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4 not found: ID does not exist" containerID="200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4" Jul 27 22:28:10.113378 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.113337 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4"} err="failed to get container status \"200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4\": rpc error: code = NotFound desc = could not find container \"200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4\": container with ID starting with 200d8a883748a53f19f28fd46cea7640ebb917eb51242324057478ceac37aba4 not found: ID does not exist" Jul 27 22:28:10.113378 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.113357 2556 scope.go:117] "RemoveContainer" containerID="749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b" Jul 27 22:28:10.113600 ip-10-0-133-37 kubenswrapper[2556]: E0727 22:28:10.113581 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b\": container with ID starting with 749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b not found: ID does not exist" containerID="749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b" Jul 27 22:28:10.113642 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.113607 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b"} err="failed to get container status \"749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b\": rpc error: code = NotFound desc = could not find container \"749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b\": container with ID starting with 749939a2d2865b54cf091d7b9664fcdd641c4dbf3c82950483affb6f4dfaad4b not found: ID does not exist" Jul 27 22:28:10.125909 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.125874 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77"] Jul 27 22:28:10.131447 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.131418 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd2178-openshift-default-57fb8f8dfd-gpx77"] Jul 27 22:28:10.192482 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:10.192445 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" path="/var/lib/kubelet/pods/5b6150c9-f9f0-4157-a504-b616434d341a/volumes" Jul 27 22:28:12.681857 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:12.681809 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:12.781039 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:12.781005 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:12.878116 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:12.878073 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:12.980945 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:12.980905 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:13.553108 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:13.553066 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:15.697987 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:15.697950 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:28:20.706184 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:20.706129 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:20.812125 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:20.812085 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:20.973528 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:20.973424 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx"] Jul 27 22:28:20.973719 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:20.973665 2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" containerID="cri-o://f10ac6ec2fd0aa752bb6f6a6b44b6acb92707d80bbc5e1ce2626538ce8918fca" gracePeriod=30 Jul 27 22:28:21.000779 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:21.000740 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:21.234069 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:21.233975 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:26.183835 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.183804 2556 generic.go:356] "Generic (PLEG): container finished" podID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerID="f10ac6ec2fd0aa752bb6f6a6b44b6acb92707d80bbc5e1ce2626538ce8918fca" exitCode=0 Jul 27 22:28:26.184296 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.183872 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" event={"ID":"47410e32-b2fd-4d9a-a161-78f3410ea164","Type":"ContainerDied","Data":"f10ac6ec2fd0aa752bb6f6a6b44b6acb92707d80bbc5e1ce2626538ce8918fca"} Jul 27 22:28:26.184296 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.183943 2556 scope.go:117] "RemoveContainer" containerID="b291930fe5972f5f5bdd8dcdbff11629da22e9d7008b108f4ebb0cdee1e82697" Jul 27 22:28:26.225892 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.225866 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:28:26.335855 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.335757 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-certs\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.335855 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.335818 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-socket\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.335855 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.335854 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47410e32-b2fd-4d9a-a161-78f3410ea164-istiod-ca-cert\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.336153 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.335875 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-podinfo\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.336153 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.335897 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-data\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.336153 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.335913 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-token\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.336153 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.335978 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-credential-socket\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.336153 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336010 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-envoy\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.336153 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336053 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fql6\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-kube-api-access-9fql6\") pod \"47410e32-b2fd-4d9a-a161-78f3410ea164\" (UID: \"47410e32-b2fd-4d9a-a161-78f3410ea164\") " Jul 27 22:28:26.336153 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336076 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:26.336758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336221 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:26.336758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336313 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:26.336758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336338 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47410e32-b2fd-4d9a-a161-78f3410ea164-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 22:28:26.336758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336410 2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-certs\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.336758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336429 2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-workload-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.336758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336443 2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47410e32-b2fd-4d9a-a161-78f3410ea164-istiod-ca-cert\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.336758 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336455 2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-credential-socket\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.337386 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.336757 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-data" (OuterVolumeSpecName: "istio-data") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:26.338507 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.338475 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 22:28:26.338635 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.338529 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-kube-api-access-9fql6" (OuterVolumeSpecName: "kube-api-access-9fql6") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "kube-api-access-9fql6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:28:26.338830 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.338803 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-token" (OuterVolumeSpecName: "istio-token") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 22:28:26.338923 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.338849 2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47410e32-b2fd-4d9a-a161-78f3410ea164" (UID: "47410e32-b2fd-4d9a-a161-78f3410ea164"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 22:28:26.437902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.437866 2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-podinfo\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.437902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.437895 2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-data\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.437902 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.437904 2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-token\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.438142 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.437913 2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47410e32-b2fd-4d9a-a161-78f3410ea164-istio-envoy\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:26.438142 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:26.437922 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fql6\" (UniqueName: \"kubernetes.io/projected/47410e32-b2fd-4d9a-a161-78f3410ea164-kube-api-access-9fql6\") on node \"ip-10-0-133-37.ec2.internal\" DevicePath \"\"" Jul 27 22:28:27.190525 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:27.190490 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" Jul 27 22:28:27.190525 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:27.190511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx" event={"ID":"47410e32-b2fd-4d9a-a161-78f3410ea164","Type":"ContainerDied","Data":"9c62f8f35a44016ae3f9798e444c0d8f166cd643e06ff18f9e88fd92715febc9"} Jul 27 22:28:27.191117 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:27.190569 2556 scope.go:117] "RemoveContainer" containerID="f10ac6ec2fd0aa752bb6f6a6b44b6acb92707d80bbc5e1ce2626538ce8918fca" Jul 27 22:28:27.214406 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:27.214377 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx"] Jul 27 22:28:27.219334 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:27.219311 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-54963f-openshift-default-57d6974c8-gxlxx"] Jul 27 22:28:28.193348 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:28.193305 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" path="/var/lib/kubelet/pods/47410e32-b2fd-4d9a-a161-78f3410ea164/volumes" Jul 27 22:28:28.513748 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:28.513717 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:28.615446 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:28.615410 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:30.993036 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:30.992998 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:28:35.288472 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:35.288438 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:35.393569 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:35.393529 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:35.595810 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:35.595717 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:35.848966 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:35.848878 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:39.019464 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:39.019412 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:41.191422 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:41.191379 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:41.274871 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:41.274840 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:41.347744 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:41.347706 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:41.440976 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:41.440940 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:41.857177 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:41.857142 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:43.001897 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:43.001861 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vvwp4"] Jul 27 22:28:48.256889 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:48.256851 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:48.350763 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:48.350728 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:48.546756 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:48.546645 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:48.741561 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:48.741524 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:28:54.714276 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:54.714236 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w8mt6_3b878611-46b4-4b23-a882-11dcbc016cfa/manager/0.log" Jul 27 22:28:54.838710 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:54.838665 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wqkhh_2e6e129f-313e-499a-80c5-a623d02b30f0/manager/0.log" Jul 27 22:28:55.093033 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:55.092941 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vqssk_f3cfb657-58c7-4501-aa5b-4883ee907a5e/server/0.log" Jul 27 22:28:55.226048 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:55.226017 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qhx8r_5ca527b1-69df-4373-94cc-e0a164209688/manager/0.log" Jul 27 22:28:55.351375 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:55.351299 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d57c545b5-57d46_83de5dde-f09f-438f-8dc8-74889a1429ae/copy-manifests/0.log" Jul 27 22:28:55.366609 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:55.366583 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d57c545b5-57d46_83de5dde-f09f-438f-8dc8-74889a1429ae/manager/0.log" Jul 27 22:28:57.255303 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.255268 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk_4d37d026-6f4b-407e-9f9c-a65fb2fa777b/util/0.log" Jul 27 22:28:57.262734 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.262706 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk_4d37d026-6f4b-407e-9f9c-a65fb2fa777b/pull/0.log" Jul 27 22:28:57.270405 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.270382 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk_4d37d026-6f4b-407e-9f9c-a65fb2fa777b/extract/0.log" Jul 27 22:28:57.385486 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.385458 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4_b0004e05-a625-4091-bcdc-e21f0646f7e3/util/0.log" Jul 27 22:28:57.393603 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.393576 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4_b0004e05-a625-4091-bcdc-e21f0646f7e3/pull/0.log" Jul 27 22:28:57.403000 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.402976 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4_b0004e05-a625-4091-bcdc-e21f0646f7e3/extract/0.log" Jul 27 22:28:57.514648 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.514560 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr_3290fdcd-a808-48ba-8d1f-be7e851e9546/util/0.log" Jul 27 22:28:57.523216 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.523185 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr_3290fdcd-a808-48ba-8d1f-be7e851e9546/pull/0.log" Jul 27 22:28:57.530854 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.530833 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr_3290fdcd-a808-48ba-8d1f-be7e851e9546/extract/0.log" Jul 27 22:28:57.646635 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.646600 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8_991508b7-4c79-4b6b-98eb-aa26853b3758/util/0.log" Jul 27 22:28:57.653737 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.653716 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8_991508b7-4c79-4b6b-98eb-aa26853b3758/pull/0.log" Jul 27 22:28:57.661010 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.660982 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8_991508b7-4c79-4b6b-98eb-aa26853b3758/extract/0.log" Jul 27 22:28:57.912690 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:57.912580 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mgvg8_7eaed7ec-8d14-44c8-b611-01c31323f2ab/manager/0.log" Jul 27 22:28:58.025592 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:58.025564 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pdzgx_2b404ec0-9d1a-40b8-9e41-549f4f4b5b59/manager/0.log" Jul 27 22:28:58.140925 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:58.140900 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vxtpz_34cc2ea3-47d9-4416-bea6-72011567f324/kuadrant-console-plugin/0.log" Jul 27 22:28:58.266875 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:58.266841 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wl4x4_f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f/registry-server/0.log" Jul 27 22:28:58.505222 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:58.505188 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vvwp4_55aad1c8-4a1b-425f-94ed-0842980f3a83/limitador/0.log" Jul 27 22:28:58.631682 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:58.631576 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kh6jg_41464d90-b6cb-461e-8ecb-89de3caf27dc/manager/0.log" Jul 27 22:28:58.995970 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:58.995936 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c66lck_18a1ecc8-7323-4d14-84e6-e763daf75cb4/istio-proxy/0.log" Jul 27 22:28:59.232053 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:59.232022 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd65f9cdf-8tcdx_4a92de9c-6da0-41be-908f-ba23b2de00ba/kube-auth-proxy/0.log" Jul 27 22:28:59.467690 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:28:59.467638 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xp6zp_db631f7d-1a1a-4446-a872-9c91fb4270bf/istio-proxy/1.log" Jul 27 22:29:00.671537 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:00.671504 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-97bzt_2ea6bc5c-e748-4bd6-9ca7-118375dc8dbd/main/0.log" Jul 27 22:29:00.793381 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:00.793348 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fzh9d_c3a3f640-b2a6-4d63-9b3a-efea5401f9e1/main/0.log" Jul 27 22:29:06.284910 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:06.284872 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:06.379243 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:06.379205 2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" limit=1024 Jul 27 22:29:09.326439 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:09.326412 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m24q9_8093b04e-e222-499b-b2e2-8bc7b0fc351d/global-pull-secret-syncer/0.log" Jul 27 22:29:09.495877 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:09.495842 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fmvdv_7347070d-ff62-4d14-9b12-433decb4d5b6/konnectivity-agent/0.log" Jul 27 22:29:09.643526 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:09.643401 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-37.ec2.internal_0768a6ccd589953a80216109a3092621/haproxy/0.log" Jul 27 22:29:13.836849 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:13.836763 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk_4d37d026-6f4b-407e-9f9c-a65fb2fa777b/extract/0.log" Jul 27 22:29:13.867605 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:13.867569 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk_4d37d026-6f4b-407e-9f9c-a65fb2fa777b/util/0.log" Jul 27 22:29:13.893073 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:13.893041 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xtfwk_4d37d026-6f4b-407e-9f9c-a65fb2fa777b/pull/0.log" Jul 27 22:29:13.927823 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:13.927791 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4_b0004e05-a625-4091-bcdc-e21f0646f7e3/extract/0.log" Jul 27 22:29:13.964206 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:13.964177 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4_b0004e05-a625-4091-bcdc-e21f0646f7e3/util/0.log" Jul 27 22:29:13.994275 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:13.994246 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c4nn4_b0004e05-a625-4091-bcdc-e21f0646f7e3/pull/0.log" Jul 27 22:29:14.024341 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.024316 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr_3290fdcd-a808-48ba-8d1f-be7e851e9546/extract/0.log" Jul 27 22:29:14.051799 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.051760 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr_3290fdcd-a808-48ba-8d1f-be7e851e9546/util/0.log" Jul 27 22:29:14.076082 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.076047 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hdxnr_3290fdcd-a808-48ba-8d1f-be7e851e9546/pull/0.log" Jul 27 22:29:14.108853 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.108769 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8_991508b7-4c79-4b6b-98eb-aa26853b3758/extract/0.log" Jul 27 22:29:14.139244 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.139214 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8_991508b7-4c79-4b6b-98eb-aa26853b3758/util/0.log" Jul 27 22:29:14.164443 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.164406 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef158vk8_991508b7-4c79-4b6b-98eb-aa26853b3758/pull/0.log" Jul 27 22:29:14.711597 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.711568 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mgvg8_7eaed7ec-8d14-44c8-b611-01c31323f2ab/manager/0.log" Jul 27 22:29:14.744977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.744952 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pdzgx_2b404ec0-9d1a-40b8-9e41-549f4f4b5b59/manager/0.log" Jul 27 22:29:14.776236 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.776211 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vxtpz_34cc2ea3-47d9-4416-bea6-72011567f324/kuadrant-console-plugin/0.log" Jul 27 22:29:14.836978 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:14.836942 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wl4x4_f7ad1d69-8928-4d7f-bdc4-b9ed2001e19f/registry-server/0.log" Jul 27 22:29:15.176630 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:15.176541 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vvwp4_55aad1c8-4a1b-425f-94ed-0842980f3a83/limitador/0.log" Jul 27 22:29:15.286803 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:15.286772 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kh6jg_41464d90-b6cb-461e-8ecb-89de3caf27dc/manager/0.log" Jul 27 22:29:17.459070 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:17.459034 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9f44c_4b1902dc-5464-4ef7-9e83-b6a74d10a929/node-exporter/0.log" Jul 27 22:29:17.498844 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:17.498812 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9f44c_4b1902dc-5464-4ef7-9e83-b6a74d10a929/kube-rbac-proxy/0.log" Jul 27 22:29:17.530727 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:17.530703 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9f44c_4b1902dc-5464-4ef7-9e83-b6a74d10a929/init-textfile/0.log" Jul 27 22:29:19.769450 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.769409 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc"] Jul 27 22:29:19.769977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.769932 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" Jul 27 22:29:19.769977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.769951 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" Jul 27 22:29:19.769977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.769964 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" Jul 27 22:29:19.769977 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.769972 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.769996 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770005 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770021 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" containerName="main" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770029 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" containerName="main" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770045 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770053 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770173 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770187 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6150c9-f9f0-4157-a504-b616434d341a" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770198 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770209 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="47410e32-b2fd-4d9a-a161-78f3410ea164" containerName="istio-proxy" Jul 27 22:29:19.770221 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.770219 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7df93f4-fe5f-4c0b-bab2-a759f5ece2fb" containerName="main" Jul 27 22:29:19.773005 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.772982 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:19.776058 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.776039 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j99rk\"/\"openshift-service-ca.crt\"" Jul 27 22:29:19.776135 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.776039 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j99rk\"/\"kube-root-ca.crt\"" Jul 27 22:29:19.776973 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.776953 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j99rk\"/\"default-dockercfg-f5phm\"" Jul 27 22:29:19.791871 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.791845 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc"] Jul 27 22:29:19.912306 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.912272 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-sys\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:19.912505 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.912314 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-lib-modules\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:19.912505 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.912421 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-podres\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:19.912612 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.912505 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqs4k\" (UniqueName: \"kubernetes.io/projected/c126335e-7c04-4858-a524-535085decb68-kube-api-access-xqs4k\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:19.912612 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:19.912528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-proc\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013087 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013049 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-sys\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013087 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-lib-modules\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013118 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-podres\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013156 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqs4k\" (UniqueName: \"kubernetes.io/projected/c126335e-7c04-4858-a524-535085decb68-kube-api-access-xqs4k\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013175 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-proc\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013197 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-sys\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013240 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-lib-modules\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013335 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013296 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-podres\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.013535 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.013338 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c126335e-7c04-4858-a524-535085decb68-proc\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.023820 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.023757 2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqs4k\" (UniqueName: \"kubernetes.io/projected/c126335e-7c04-4858-a524-535085decb68-kube-api-access-xqs4k\") pod \"perf-node-gather-daemonset-8pjlc\" (UID: \"c126335e-7c04-4858-a524-535085decb68\") " pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.082619 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.082569 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.224519 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.224323 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc"] Jul 27 22:29:20.226814 ip-10-0-133-37 kubenswrapper[2556]: W0727 22:29:20.226781 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc126335e_7c04_4858_a524_535085decb68.slice/crio-a93a67ebe4c2c7e0975a32a9e83b8b32e927b6249eed0305f03cf1066cf34372 WatchSource:0}: Error finding container a93a67ebe4c2c7e0975a32a9e83b8b32e927b6249eed0305f03cf1066cf34372: Status 404 returned error can't find the container with id a93a67ebe4c2c7e0975a32a9e83b8b32e927b6249eed0305f03cf1066cf34372 Jul 27 22:29:20.478165 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.478120 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" event={"ID":"c126335e-7c04-4858-a524-535085decb68","Type":"ContainerStarted","Data":"e084ae8b1f7db293f83310fd614e6e66695f518ec011f934774f2011df36f18f"} Jul 27 22:29:20.478342 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.478184 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" event={"ID":"c126335e-7c04-4858-a524-535085decb68","Type":"ContainerStarted","Data":"a93a67ebe4c2c7e0975a32a9e83b8b32e927b6249eed0305f03cf1066cf34372"} Jul 27 22:29:20.478342 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.478226 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:20.497586 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:20.497537 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" podStartSLOduration=1.497523878 podStartE2EDuration="1.497523878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 22:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:29:20.495486702 +0000 UTC m=+3328.903908732" watchObservedRunningTime="2026-07-27 22:29:20.497523878 +0000 UTC m=+3328.905945885" Jul 27 22:29:22.706881 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:22.706852 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tfdcz_e4d5c4d8-db45-4169-9cfc-6368ce4c34a3/dns/0.log" Jul 27 22:29:22.736068 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:22.736042 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tfdcz_e4d5c4d8-db45-4169-9cfc-6368ce4c34a3/kube-rbac-proxy/0.log" Jul 27 22:29:22.881311 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:22.881288 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qwnv7_9a5f0961-958b-4d25-846a-6320dc6a87be/dns-node-resolver/0.log" Jul 27 22:29:23.475389 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:23.475361 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-797cb7d859-wmrhq_007c7d5a-3a89-42e7-a4aa-d24ab75177d4/registry/0.log" Jul 27 22:29:23.538064 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:23.538034 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7rkt6_edb9df30-5601-4bcc-9d00-3d531da72dd5/node-ca/0.log" Jul 27 22:29:24.554906 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:24.554875 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c66lck_18a1ecc8-7323-4d14-84e6-e763daf75cb4/istio-proxy/0.log" Jul 27 22:29:24.722234 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:24.722200 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd65f9cdf-8tcdx_4a92de9c-6da0-41be-908f-ba23b2de00ba/kube-auth-proxy/0.log" Jul 27 22:29:24.784978 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:24.784941 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xp6zp_db631f7d-1a1a-4446-a872-9c91fb4270bf/istio-proxy/0.log" Jul 27 22:29:24.838558 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:24.838473 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xp6zp_db631f7d-1a1a-4446-a872-9c91fb4270bf/istio-proxy/1.log" Jul 27 22:29:25.810014 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:25.809978 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cg7rm_8f664142-e58d-4573-8ab9-40738e1a5c70/serve-healthcheck-canary/0.log" Jul 27 22:29:26.494553 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:26.494526 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j99rk/perf-node-gather-daemonset-8pjlc" Jul 27 22:29:26.546350 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:26.546308 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zswrh_36d73831-2e20-4aab-8964-c5850d2dc0d0/kube-rbac-proxy/0.log" Jul 27 22:29:26.571762 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:26.571727 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zswrh_36d73831-2e20-4aab-8964-c5850d2dc0d0/exporter/0.log" Jul 27 22:29:26.597646 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:26.597620 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zswrh_36d73831-2e20-4aab-8964-c5850d2dc0d0/extractor/0.log" Jul 27 22:29:29.048690 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:29.048645 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w8mt6_3b878611-46b4-4b23-a882-11dcbc016cfa/manager/0.log" Jul 27 22:29:29.434183 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:29.434093 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wqkhh_2e6e129f-313e-499a-80c5-a623d02b30f0/manager/0.log" Jul 27 22:29:29.817468 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:29.817441 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vqssk_f3cfb657-58c7-4501-aa5b-4883ee907a5e/server/0.log" Jul 27 22:29:29.976163 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:29.976128 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qhx8r_5ca527b1-69df-4373-94cc-e0a164209688/manager/0.log" Jul 27 22:29:30.045561 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:30.045535 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d57c545b5-57d46_83de5dde-f09f-438f-8dc8-74889a1429ae/manager/0.log" Jul 27 22:29:30.087787 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:30.087699 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d57c545b5-57d46_83de5dde-f09f-438f-8dc8-74889a1429ae/copy-manifests/0.log" Jul 27 22:29:30.128697 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:30.128646 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c66lck_18a1ecc8-7323-4d14-84e6-e763daf75cb4/istio-proxy/0.log" Jul 27 22:29:30.300529 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:30.300500 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd65f9cdf-8tcdx_4a92de9c-6da0-41be-908f-ba23b2de00ba/kube-auth-proxy/0.log" Jul 27 22:29:30.347326 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:30.347221 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xp6zp_db631f7d-1a1a-4446-a872-9c91fb4270bf/istio-proxy/0.log" Jul 27 22:29:30.404803 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:30.404772 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xp6zp_db631f7d-1a1a-4446-a872-9c91fb4270bf/istio-proxy/1.log" Jul 27 22:29:31.635315 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:31.635279 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cn56p_afaa0467-831b-4304-b129-d4139f622133/openshift-lws-operator/0.log" Jul 27 22:29:31.874973 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:31.874938 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c66lck_18a1ecc8-7323-4d14-84e6-e763daf75cb4/istio-proxy/0.log" Jul 27 22:29:32.026086 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:32.026055 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd65f9cdf-8tcdx_4a92de9c-6da0-41be-908f-ba23b2de00ba/kube-auth-proxy/0.log" Jul 27 22:29:32.066419 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:32.066324 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xp6zp_db631f7d-1a1a-4446-a872-9c91fb4270bf/istio-proxy/0.log" Jul 27 22:29:32.120601 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:32.120557 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-xp6zp_db631f7d-1a1a-4446-a872-9c91fb4270bf/istio-proxy/1.log" Jul 27 22:29:36.534048 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:36.534016 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mqxcs_8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec/migrator/0.log" Jul 27 22:29:36.561296 ip-10-0-133-37 kubenswrapper[2556]: I0727 22:29:36.561260 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mqxcs_8ab1a4c3-f32a-4f5a-8a76-319e225ee3ec/graceful-termination/0.log"